Note: This is test shard 8 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from TableRangeRebalancingTest
[ RUN ] TableRangeRebalancingTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 08:11:06.249939 25068 test_util.cc:268] Using random seed: 1122994856
I20240503 08:11:06.261143 25068 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:06.261219 25068 ts_itest-base.cc:113] --------------
I20240503 08:11:06.261256 25068 ts_itest-base.cc:114] 3 tablet servers
I20240503 08:11:06.261315 25068 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:06.261372 25068 ts_itest-base.cc:116] --------------
2024-05-03T08:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:06Z Disabled control of system clock
I20240503 08:11:06.282603 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:40879
--webserver_interface=127.24.123.62
--webserver_port=0
--builtin_ntp_servers=127.24.123.20:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.123.62:40879
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:11:06.592768 25080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:06.593070 25080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:06.593194 25080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:06.600211 25080 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:06.600343 25080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:06.600419 25080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:06.600495 25080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:06.600560 25080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:06.609444 25080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.123.62:40879
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:40879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.123.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:06.610641 25080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:06.612299 25080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:06.623147 25086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.623514 25085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.624245 25088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:06.624747 25080 server_base.cc:1034] running on GCE node
I20240503 08:11:06.625406 25080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:06.626690 25080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:06.627874 25080 hybrid_clock.cc:648] HybridClock initialized: now 1714723866627827 us; error 48 us; skew 500 ppm
I20240503 08:11:06.628162 25080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:06.634436 25080 webserver.cc:460] Webserver started at http://127.24.123.62:43621/ using document root <none> and password file <none>
I20240503 08:11:06.635159 25080 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:06.635249 25080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:06.635502 25080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:06.637866 25080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-gkrw"
I20240503 08:11:06.638532 25080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-gkrw"
I20240503 08:11:06.643718 25080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 08:11:06.646751 25094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.647939 25080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:11:06.648083 25080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-gkrw"
I20240503 08:11:06.648248 25080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:06.661203 25080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:06.661911 25080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:06.662150 25080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:06.680907 25080 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.62:40879
I20240503 08:11:06.680923 25145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.62:40879 every 8 connection(s)
I20240503 08:11:06.682103 25080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:06.684947 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 25080
I20240503 08:11:06.685156 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:06.685451 25146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:06.695040 25146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95: Bootstrap starting.
I20240503 08:11:06.697676 25146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:06.698565 25146 log.cc:824] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:06.700929 25146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95: No bootstrap required, opened a new log
I20240503 08:11:06.706977 25146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40879 } }
I20240503 08:11:06.707247 25146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:06.707336 25146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9faec04b78ee4e4eac6b1f0acc9e5f95, State: Initialized, Role: FOLLOWER
I20240503 08:11:06.707944 25146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40879 } }
I20240503 08:11:06.708174 25146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:06.708313 25146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:06.708484 25146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:06.710500 25146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40879 } }
I20240503 08:11:06.711035 25146 leader_election.cc:302] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9faec04b78ee4e4eac6b1f0acc9e5f95; no voters:
I20240503 08:11:06.711596 25146 leader_election.cc:288] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:06.711697 25151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:06.712203 25151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [term 1 LEADER]: Becoming Leader. State: Replica: 9faec04b78ee4e4eac6b1f0acc9e5f95, State: Running, Role: LEADER
I20240503 08:11:06.712695 25151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40879 } }
I20240503 08:11:06.713207 25146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:06.715392 25153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9faec04b78ee4e4eac6b1f0acc9e5f95. Latest consensus state: current_term: 1 leader_uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40879 } } }
I20240503 08:11:06.715399 25152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faec04b78ee4e4eac6b1f0acc9e5f95" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40879 } } }
I20240503 08:11:06.715790 25152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:06.715823 25153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:06.716846 25156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:06.721959 25156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:06.729522 25156 catalog_manager.cc:1347] Generated new cluster ID: 4b78bbf2446f4003b83ea269e295dcfb
I20240503 08:11:06.729620 25156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:06.742553 25156 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:06.744174 25156 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:06.762373 25156 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9faec04b78ee4e4eac6b1f0acc9e5f95: Generated new TSK 0
I20240503 08:11:06.763413 25156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:06.773165 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.1:0
--local_ip_for_outbound_sockets=127.24.123.1
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40879
--builtin_ntp_servers=127.24.123.20:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:07.094115 25170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.094403 25170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.094529 25170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.102202 25170 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:07.102342 25170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.102494 25170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.1
I20240503 08:11:07.111251 25170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.1
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.112531 25170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.114208 25170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.125988 25176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.126484 25175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.127022 25178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.127723 25170 server_base.cc:1034] running on GCE node
I20240503 08:11:07.128243 25170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.129124 25170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.130302 25170 hybrid_clock.cc:648] HybridClock initialized: now 1714723867130243 us; error 52 us; skew 500 ppm
I20240503 08:11:07.130604 25170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.133186 25170 webserver.cc:460] Webserver started at http://127.24.123.1:43083/ using document root <none> and password file <none>
I20240503 08:11:07.133872 25170 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.133970 25170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.134263 25170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.136981 25170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9250c2e966374fc8b65d634d97d41c11"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-gkrw"
I20240503 08:11:07.138156 25170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9250c2e966374fc8b65d634d97d41c11"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-gkrw"
I20240503 08:11:07.142921 25170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240503 08:11:07.145959 25184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.147217 25170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 08:11:07.147342 25170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9250c2e966374fc8b65d634d97d41c11"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-gkrw"
I20240503 08:11:07.147498 25170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.160866 25170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.161576 25170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:07.161795 25170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.162604 25170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.163969 25170 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.164054 25170 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.164130 25170 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.164181 25170 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.194013 25170 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.1:36251
I20240503 08:11:07.194057 25296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.1:36251 every 8 connection(s)
I20240503 08:11:07.195297 25170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:07.199154 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 25170
I20240503 08:11:07.199374 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:07.202467 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.2:0
--local_ip_for_outbound_sockets=127.24.123.2
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40879
--builtin_ntp_servers=127.24.123.20:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:07.208165 25297 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40879
I20240503 08:11:07.208492 25297 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.209455 25297 heartbeater.cc:511] Master 127.24.123.62:40879 requested a full tablet report, sending...
I20240503 08:11:07.211586 25111 ts_manager.cc:194] Registered new tserver with Master: 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251)
I20240503 08:11:07.213119 25111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.1:53447
W20240503 08:11:07.521011 25300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.521275 25300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.521404 25300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.528537 25300 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:07.528651 25300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.528797 25300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.2
I20240503 08:11:07.537770 25300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.2
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.539023 25300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.540637 25300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.552373 25305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.553025 25306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.553551 25308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.554221 25300 server_base.cc:1034] running on GCE node
I20240503 08:11:07.554759 25300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.555681 25300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.556887 25300 hybrid_clock.cc:648] HybridClock initialized: now 1714723867556840 us; error 38 us; skew 500 ppm
I20240503 08:11:07.557199 25300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.559556 25300 webserver.cc:460] Webserver started at http://127.24.123.2:39023/ using document root <none> and password file <none>
I20240503 08:11:07.560237 25300 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.560338 25300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.560606 25300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.563143 25300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "baf64abf9ec642e0acb49eb700e54205"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-gkrw"
I20240503 08:11:07.563864 25300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "baf64abf9ec642e0acb49eb700e54205"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-gkrw"
I20240503 08:11:07.568681 25300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:11:07.571797 25314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.573417 25300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:07.573544 25300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "baf64abf9ec642e0acb49eb700e54205"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-gkrw"
I20240503 08:11:07.573730 25300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.588230 25300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.588995 25300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:07.589244 25300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.590098 25300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.591974 25300 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.592053 25300 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.592155 25300 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.592218 25300 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.623013 25300 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.2:32849
I20240503 08:11:07.623061 25426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.2:32849 every 8 connection(s)
I20240503 08:11:07.624258 25300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:07.628628 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 25300
I20240503 08:11:07.628778 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:07.631644 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.3:0
--local_ip_for_outbound_sockets=127.24.123.3
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40879
--builtin_ntp_servers=127.24.123.20:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:07.637287 25427 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40879
I20240503 08:11:07.637624 25427 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.638545 25427 heartbeater.cc:511] Master 127.24.123.62:40879 requested a full tablet report, sending...
I20240503 08:11:07.639945 25111 ts_manager.cc:194] Registered new tserver with Master: baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:07.640682 25111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.2:50221
W20240503 08:11:07.953171 25430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.953442 25430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.953563 25430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.961447 25430 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:07.961561 25430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.961702 25430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.3
I20240503 08:11:07.970408 25430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.3
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.971709 25430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.973330 25430 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.985483 25435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.985982 25436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.986541 25438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.987200 25430 server_base.cc:1034] running on GCE node
I20240503 08:11:07.987800 25430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.988811 25430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.990025 25430 hybrid_clock.cc:648] HybridClock initialized: now 1714723867989976 us; error 41 us; skew 500 ppm
I20240503 08:11:07.990341 25430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.992499 25430 webserver.cc:460] Webserver started at http://127.24.123.3:43573/ using document root <none> and password file <none>
I20240503 08:11:07.993204 25430 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.993317 25430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.993611 25430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.996623 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "89d5514a6364453581ede69f4179186b"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-gkrw"
I20240503 08:11:07.997340 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "89d5514a6364453581ede69f4179186b"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-gkrw"
I20240503 08:11:08.002125 25430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 08:11:08.005352 25444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.006582 25430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:08.006717 25430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "89d5514a6364453581ede69f4179186b"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-gkrw"
I20240503 08:11:08.006927 25430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:08.045272 25430 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:08.046149 25430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:08.046427 25430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:08.047376 25430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:08.048811 25430 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:08.048902 25430 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.049007 25430 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:08.049073 25430 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.079802 25430 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.3:44695
I20240503 08:11:08.079865 25556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.3:44695 every 8 connection(s)
I20240503 08:11:08.081035 25430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:08.088279 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 25430
I20240503 08:11:08.088474 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:08.093361 25557 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40879
I20240503 08:11:08.093644 25557 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:08.094400 25557 heartbeater.cc:511] Master 127.24.123.62:40879 requested a full tablet report, sending...
I20240503 08:11:08.095640 25111 ts_manager.cc:194] Registered new tserver with Master: 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:08.096375 25111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.3:41705
I20240503 08:11:08.104120 25068 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:11:08.122771 25068 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:11:08.216650 25297 heartbeater.cc:503] Master 127.24.123.62:40879 was elected leader, sending a full tablet report...
I20240503 08:11:08.643656 25427 heartbeater.cc:503] Master 127.24.123.62:40879 was elected leader, sending a full tablet report...
I20240503 08:11:09.098513 25557 heartbeater.cc:503] Master 127.24.123.62:40879 was elected leader, sending a full tablet report...
I20240503 08:11:11.876727 25111 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43298:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\200\007\001\000\000\000\000\000\006\001\000\000\000\000\000\007\001\000\377\377\377\177"
indirect_data: "<redacted>"""
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 8
seed: 0
}
}
W20240503 08:11:11.882843 25111 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:11:11.921633 25229 tablet_service.cc:1467] Processing CreateTablet for tablet 92fda691c4a54496b216ca52582b3aa2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.923252 25227 tablet_service.cc:1467] Processing CreateTablet for tablet b7a899d61a8b481eb8f02bd4f1997a05 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.923282 25361 tablet_service.cc:1467] Processing CreateTablet for tablet 286d705044a24c9b995fa8795df2205a (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.924134 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a899d61a8b481eb8f02bd4f1997a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.924767 25361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286d705044a24c9b995fa8795df2205a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.924616 25485 tablet_service.cc:1467] Processing CreateTablet for tablet 6175bd8e1b7344d086a6ad2d59b50841 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.923875 25477 tablet_service.cc:1467] Processing CreateTablet for tablet 81603a53190547b6b5d468d86d374c87 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.925228 25487 tablet_service.cc:1467] Processing CreateTablet for tablet b7a899d61a8b481eb8f02bd4f1997a05 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.925674 25485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6175bd8e1b7344d086a6ad2d59b50841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.925796 25489 tablet_service.cc:1467] Processing CreateTablet for tablet 92fda691c4a54496b216ca52582b3aa2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.926254 25489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fda691c4a54496b216ca52582b3aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.924257 25479 tablet_service.cc:1467] Processing CreateTablet for tablet 06530128c24242828d4a602f75b6bfe7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.926926 25479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06530128c24242828d4a602f75b6bfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.922528 25362 tablet_service.cc:1467] Processing CreateTablet for tablet 48a82bdbebae44489ec337f79c21507e (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.927076 25230 tablet_service.cc:1467] Processing CreateTablet for tablet 44dfb618fbfc4e05bd57d7b6a97bb7c3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.927307 25225 tablet_service.cc:1467] Processing CreateTablet for tablet 6175bd8e1b7344d086a6ad2d59b50841 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.927614 25230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44dfb618fbfc4e05bd57d7b6a97bb7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.927675 25360 tablet_service.cc:1467] Processing CreateTablet for tablet 44dfb618fbfc4e05bd57d7b6a97bb7c3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.927850 25225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6175bd8e1b7344d086a6ad2d59b50841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.928107 25360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44dfb618fbfc4e05bd57d7b6a97bb7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.928113 25483 tablet_service.cc:1467] Processing CreateTablet for tablet cc7fdef04645494e8569400bb5ae66ce (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.929123 25483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7fdef04645494e8569400bb5ae66ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.929064 25359 tablet_service.cc:1467] Processing CreateTablet for tablet 92fda691c4a54496b216ca52582b3aa2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.929307 25492 tablet_service.cc:1467] Processing CreateTablet for tablet 48a82bdbebae44489ec337f79c21507e (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.929472 25359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fda691c4a54496b216ca52582b3aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.929811 25492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a82bdbebae44489ec337f79c21507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.929929 25232 tablet_service.cc:1467] Processing CreateTablet for tablet 48a82bdbebae44489ec337f79c21507e (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.930402 25232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a82bdbebae44489ec337f79c21507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.930752 25358 tablet_service.cc:1467] Processing CreateTablet for tablet f7ea4abd02e24b98935c3e614c2b7de8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.930757 25490 tablet_service.cc:1467] Processing CreateTablet for tablet 44dfb618fbfc4e05bd57d7b6a97bb7c3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.931036 25481 tablet_service.cc:1467] Processing CreateTablet for tablet 2291b936dd324f0fbaa0ff23b1369447 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.931237 25358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ea4abd02e24b98935c3e614c2b7de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.932018 25357 tablet_service.cc:1467] Processing CreateTablet for tablet b7a899d61a8b481eb8f02bd4f1997a05 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.932440 25357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a899d61a8b481eb8f02bd4f1997a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.932476 25220 tablet_service.cc:1467] Processing CreateTablet for tablet ba38aecd198c4920942d189fbfe49653 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.933547 25491 tablet_service.cc:1467] Processing CreateTablet for tablet 286d705044a24c9b995fa8795df2205a (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.934123 25487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a899d61a8b481eb8f02bd4f1997a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.932948 25218 tablet_service.cc:1467] Processing CreateTablet for tablet 53f0fba475d84890a0b1496fdbe0ba28 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.934526 25362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a82bdbebae44489ec337f79c21507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.934616 25220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba38aecd198c4920942d189fbfe49653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.934983 25356 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe51bafa1a24fe0bd34f0d2333ab9b5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.935647 25218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f0fba475d84890a0b1496fdbe0ba28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.935989 25356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe51bafa1a24fe0bd34f0d2333ab9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.935322 25488 tablet_service.cc:1467] Processing CreateTablet for tablet f7ea4abd02e24b98935c3e614c2b7de8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.936816 25484 tablet_service.cc:1467] Processing CreateTablet for tablet f4d5437b33ba439f86ded3348da6159a (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.937665 25355 tablet_service.cc:1467] Processing CreateTablet for tablet 6175bd8e1b7344d086a6ad2d59b50841 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.937549 25480 tablet_service.cc:1467] Processing CreateTablet for tablet ba38aecd198c4920942d189fbfe49653 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.938135 25355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6175bd8e1b7344d086a6ad2d59b50841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.938221 25491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286d705044a24c9b995fa8795df2205a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.938120 25482 tablet_service.cc:1467] Processing CreateTablet for tablet ae79d8226b86428fa40e03c568813658 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.939198 25478 tablet_service.cc:1467] Processing CreateTablet for tablet 53f0fba475d84890a0b1496fdbe0ba28 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.940022 25486 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe51bafa1a24fe0bd34f0d2333ab9b5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.940397 25484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d5437b33ba439f86ded3348da6159a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.942658 25488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ea4abd02e24b98935c3e614c2b7de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.943567 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f0fba475d84890a0b1496fdbe0ba28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.944520 25482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae79d8226b86428fa40e03c568813658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.945922 25490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44dfb618fbfc4e05bd57d7b6a97bb7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.947064 25480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba38aecd198c4920942d189fbfe49653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.948361 25486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe51bafa1a24fe0bd34f0d2333ab9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.951223 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81603a53190547b6b5d468d86d374c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.954447 25481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2291b936dd324f0fbaa0ff23b1369447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.956902 25354 tablet_service.cc:1467] Processing CreateTablet for tablet f4d5437b33ba439f86ded3348da6159a (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.957505 25354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d5437b33ba439f86ded3348da6159a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.959118 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fda691c4a54496b216ca52582b3aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.936923 25226 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe51bafa1a24fe0bd34f0d2333ab9b5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.961958 25226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe51bafa1a24fe0bd34f0d2333ab9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.960464 25222 tablet_service.cc:1467] Processing CreateTablet for tablet ae79d8226b86428fa40e03c568813658 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.937251 25228 tablet_service.cc:1467] Processing CreateTablet for tablet f7ea4abd02e24b98935c3e614c2b7de8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.962083 25223 tablet_service.cc:1467] Processing CreateTablet for tablet cc7fdef04645494e8569400bb5ae66ce (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.965521 25353 tablet_service.cc:1467] Processing CreateTablet for tablet cc7fdef04645494e8569400bb5ae66ce (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.965382 25224 tablet_service.cc:1467] Processing CreateTablet for tablet f4d5437b33ba439f86ded3348da6159a (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.966077 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7fdef04645494e8569400bb5ae66ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.966665 25223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7fdef04645494e8569400bb5ae66ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.967757 25231 tablet_service.cc:1467] Processing CreateTablet for tablet 286d705044a24c9b995fa8795df2205a (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.968523 25231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286d705044a24c9b995fa8795df2205a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.970818 25221 tablet_service.cc:1467] Processing CreateTablet for tablet 2291b936dd324f0fbaa0ff23b1369447 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.971315 25361 tablet_service.cc:1467] Processing CreateTablet for tablet 2291b936dd324f0fbaa0ff23b1369447 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.972357 25352 tablet_service.cc:1467] Processing CreateTablet for tablet ae79d8226b86428fa40e03c568813658 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.972574 25228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ea4abd02e24b98935c3e614c2b7de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.972591 25361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2291b936dd324f0fbaa0ff23b1369447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.972756 25352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae79d8226b86428fa40e03c568813658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.973691 25222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae79d8226b86428fa40e03c568813658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.970983 25217 tablet_service.cc:1467] Processing CreateTablet for tablet 81603a53190547b6b5d468d86d374c87 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.974359 25219 tablet_service.cc:1467] Processing CreateTablet for tablet 06530128c24242828d4a602f75b6bfe7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.978075 25351 tablet_service.cc:1467] Processing CreateTablet for tablet 53f0fba475d84890a0b1496fdbe0ba28 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:11.978611 25224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d5437b33ba439f86ded3348da6159a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.978693 25351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f0fba475d84890a0b1496fdbe0ba28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.983608 25350 tablet_service.cc:1467] Processing CreateTablet for tablet 81603a53190547b6b5d468d86d374c87 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:11.984100 25350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81603a53190547b6b5d468d86d374c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.988286 25221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2291b936dd324f0fbaa0ff23b1369447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.989550 25217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81603a53190547b6b5d468d86d374c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:11.989861 25219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06530128c24242828d4a602f75b6bfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:12.005231 25356 tablet_service.cc:1467] Processing CreateTablet for tablet 06530128c24242828d4a602f75b6bfe7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 08:11:12.010131 25356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06530128c24242828d4a602f75b6bfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:12.005867 25360 tablet_service.cc:1467] Processing CreateTablet for tablet ba38aecd198c4920942d189fbfe49653 (DEFAULT_TABLE table=rebalance_test_table_0 [id=16a2fe8fdd54414aab6f0ccdfedb592b]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 08:11:12.011358 25360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba38aecd198c4920942d189fbfe49653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:12.019282 25575 tablet_bootstrap.cc:492] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.024376 25575 tablet_bootstrap.cc:654] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.025390 25575 log.cc:824] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:12.027837 25575 tablet_bootstrap.cc:492] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.028071 25575 ts_tablet_manager.cc:1367] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240503 08:11:12.036082 25575 raft_consensus.cc:348] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.036635 25575 raft_consensus.cc:374] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.036763 25575 raft_consensus.cc:729] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.039026 25575 consensus_queue.cc:260] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.040683 25578 tablet_bootstrap.cc:492] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.043388 25576 tablet_bootstrap.cc:492] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.044450 25578 tablet_bootstrap.cc:654] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.045800 25578 log.cc:824] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:12.046622 25576 tablet_bootstrap.cc:654] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.048471 25576 log.cc:824] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:12.057768 25578 tablet_bootstrap.cc:492] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.058100 25578 ts_tablet_manager.cc:1367] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.002s
I20240503 08:11:12.062304 25575 ts_tablet_manager.cc:1398] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.034s user 0.019s sys 0.000s
I20240503 08:11:12.062806 25575 tablet_bootstrap.cc:492] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.064419 25575 tablet_bootstrap.cc:654] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.067085 25576 tablet_bootstrap.cc:492] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.067363 25576 ts_tablet_manager.cc:1367] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.011s
I20240503 08:11:12.067701 25575 tablet_bootstrap.cc:492] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.067826 25575 ts_tablet_manager.cc:1367] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:11:12.068511 25575 raft_consensus.cc:348] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.068671 25575 raft_consensus.cc:374] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.068782 25575 raft_consensus.cc:729] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.068650 25578 raft_consensus.cc:348] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.069063 25578 raft_consensus.cc:374] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.068987 25575 consensus_queue.cc:260] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.069155 25578 raft_consensus.cc:729] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.069876 25575 ts_tablet_manager.cc:1398] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:12.069883 25578 consensus_queue.cc:260] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.070142 25575 tablet_bootstrap.cc:492] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.071753 25575 tablet_bootstrap.cc:654] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.076560 25576 raft_consensus.cc:348] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.077018 25576 raft_consensus.cc:374] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.077123 25576 raft_consensus.cc:729] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.077769 25576 consensus_queue.cc:260] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.078032 25578 ts_tablet_manager.cc:1398] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.020s user 0.013s sys 0.006s
I20240503 08:11:12.078445 25578 tablet_bootstrap.cc:492] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.080188 25578 tablet_bootstrap.cc:654] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.085913 25578 tablet_bootstrap.cc:492] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.086024 25576 ts_tablet_manager.cc:1398] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20240503 08:11:12.086098 25578 ts_tablet_manager.cc:1367] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 08:11:12.086437 25576 tablet_bootstrap.cc:492] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.087070 25578 raft_consensus.cc:348] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.087337 25578 raft_consensus.cc:374] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.087460 25578 raft_consensus.cc:729] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.087706 25578 consensus_queue.cc:260] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.088099 25576 tablet_bootstrap.cc:654] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.088968 25575 tablet_bootstrap.cc:492] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.089146 25575 ts_tablet_manager.cc:1367] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.007s
I20240503 08:11:12.089749 25578 ts_tablet_manager.cc:1398] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:12.090049 25578 tablet_bootstrap.cc:492] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.090541 25575 raft_consensus.cc:348] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.090760 25575 raft_consensus.cc:374] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.090818 25575 raft_consensus.cc:729] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.091149 25575 consensus_queue.cc:260] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.092308 25575 ts_tablet_manager.cc:1398] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:11:12.093046 25578 tablet_bootstrap.cc:654] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.093168 25575 tablet_bootstrap.cc:492] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.094696 25575 tablet_bootstrap.cc:654] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.098003 25578 tablet_bootstrap.cc:492] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.098146 25578 ts_tablet_manager.cc:1367] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240503 08:11:12.098269 25576 tablet_bootstrap.cc:492] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.098408 25576 ts_tablet_manager.cc:1367] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240503 08:11:12.098600 25575 tablet_bootstrap.cc:492] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.098733 25575 ts_tablet_manager.cc:1367] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240503 08:11:12.098845 25578 raft_consensus.cc:348] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.099092 25578 raft_consensus.cc:374] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.099165 25578 raft_consensus.cc:729] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.099256 25576 raft_consensus.cc:348] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.099449 25576 raft_consensus.cc:374] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.099417 25575 raft_consensus.cc:348] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.099417 25578 consensus_queue.cc:260] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.099546 25576 raft_consensus.cc:729] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.099606 25575 raft_consensus.cc:374] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.099722 25575 raft_consensus.cc:729] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.099740 25576 consensus_queue.cc:260] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.099934 25575 consensus_queue.cc:260] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.100281 25576 ts_tablet_manager.cc:1398] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:12.100575 25575 ts_tablet_manager.cc:1398] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:12.100610 25576 tablet_bootstrap.cc:492] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.100852 25575 tablet_bootstrap.cc:492] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.101787 25578 ts_tablet_manager.cc:1398] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:12.102139 25578 tablet_bootstrap.cc:492] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.102852 25575 tablet_bootstrap.cc:654] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.102931 25576 tablet_bootstrap.cc:654] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.103648 25578 tablet_bootstrap.cc:654] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.104627 25585 raft_consensus.cc:482] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.104859 25585 raft_consensus.cc:504] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.107482 25585 leader_election.cc:288] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
W20240503 08:11:12.129395 25428 tablet.cc:2369] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:12.131580 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44dfb618fbfc4e05bd57d7b6a97bb7c3" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.132195 25382 raft_consensus.cc:2454] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 0.
I20240503 08:11:12.133074 25447 leader_election.cc:302] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.133594 25585 raft_consensus.cc:2781] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.133733 25585 raft_consensus.cc:482] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.133822 25585 raft_consensus.cc:3037] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.138231 25585 raft_consensus.cc:504] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.139115 25585 leader_election.cc:288] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:12.139809 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44dfb618fbfc4e05bd57d7b6a97bb7c3" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205"
I20240503 08:11:12.140014 25382 raft_consensus.cc:3037] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.143369 25382 raft_consensus.cc:2454] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 1.
I20240503 08:11:12.143983 25447 leader_election.cc:302] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.144374 25585 raft_consensus.cc:2781] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.147109 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44dfb618fbfc4e05bd57d7b6a97bb7c3" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.147615 25251 raft_consensus.cc:3037] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.149626 25585 raft_consensus.cc:686] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [term 1 LEADER]: Becoming Leader. State: Replica: 89d5514a6364453581ede69f4179186b, State: Running, Role: LEADER
I20240503 08:11:12.150133 25585 consensus_queue.cc:237] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.150782 25251 raft_consensus.cc:2454] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 1.
I20240503 08:11:12.151216 25252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44dfb618fbfc4e05bd57d7b6a97bb7c3" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.151485 25252 raft_consensus.cc:2362] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 89d5514a6364453581ede69f4179186b in term 1. Re-sending same reply.
I20240503 08:11:12.155179 25578 tablet_bootstrap.cc:492] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.155318 25578 ts_tablet_manager.cc:1367] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.031s
I20240503 08:11:12.156052 25578 raft_consensus.cc:348] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.156391 25578 raft_consensus.cc:374] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.156512 25578 raft_consensus.cc:729] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.156759 25578 consensus_queue.cc:260] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.157366 25576 tablet_bootstrap.cc:492] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.157472 25578 ts_tablet_manager.cc:1398] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:12.157562 25576 ts_tablet_manager.cc:1367] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.017s
I20240503 08:11:12.157764 25578 tablet_bootstrap.cc:492] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.157429 25111 catalog_manager.cc:5484] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b reported cstate change: term changed from 0 to 1, leader changed from <none> to 89d5514a6364453581ede69f4179186b (127.24.123.3). New cstate: current_term: 1 leader_uuid: "89d5514a6364453581ede69f4179186b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:12.158332 25576 raft_consensus.cc:348] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.158531 25576 raft_consensus.cc:374] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.158602 25576 raft_consensus.cc:729] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.158813 25576 consensus_queue.cc:260] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.159333 25576 ts_tablet_manager.cc:1398] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:12.159394 25578 tablet_bootstrap.cc:654] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.159615 25576 tablet_bootstrap.cc:492] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.161192 25576 tablet_bootstrap.cc:654] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.161820 25575 tablet_bootstrap.cc:492] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.162027 25575 ts_tablet_manager.cc:1367] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.061s user 0.015s sys 0.038s
I20240503 08:11:12.162809 25575 raft_consensus.cc:348] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.163061 25575 raft_consensus.cc:374] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.163162 25575 raft_consensus.cc:729] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.163245 25583 raft_consensus.cc:482] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.163344 25576 tablet_bootstrap.cc:492] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.163460 25576 ts_tablet_manager.cc:1367] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 08:11:12.163424 25575 consensus_queue.cc:260] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.163905 25575 ts_tablet_manager.cc:1398] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:12.163403 25583 raft_consensus.cc:504] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.164175 25575 tablet_bootstrap.cc:492] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.164530 25576 raft_consensus.cc:348] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.164700 25576 raft_consensus.cc:374] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.164777 25576 raft_consensus.cc:729] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.164992 25576 consensus_queue.cc:260] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.165411 25575 tablet_bootstrap.cc:654] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.165753 25576 ts_tablet_manager.cc:1398] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:12.165762 25583 leader_election.cc:288] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.165992 25576 tablet_bootstrap.cc:492] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.167430 25576 tablet_bootstrap.cc:654] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.176787 25576 tablet_bootstrap.cc:492] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.177067 25576 ts_tablet_manager.cc:1367] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:11:12.178033 25576 raft_consensus.cc:348] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.178359 25576 raft_consensus.cc:374] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.178555 25576 raft_consensus.cc:729] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.178925 25576 consensus_queue.cc:260] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.179630 25576 ts_tablet_manager.cc:1398] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:12.179967 25576 tablet_bootstrap.cc:492] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.181713 25576 tablet_bootstrap.cc:654] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.185283 25512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81603a53190547b6b5d468d86d374c87" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
W20240503 08:11:12.186622 25188 leader_election.cc:341] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89d5514a6364453581ede69f4179186b (127.24.123.3:44695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.188565 25575 tablet_bootstrap.cc:492] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.188490 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81603a53190547b6b5d468d86d374c87" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.188728 25575 ts_tablet_manager.cc:1367] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.010s
I20240503 08:11:12.188753 25382 raft_consensus.cc:2454] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 0.
I20240503 08:11:12.189177 25187 leader_election.cc:302] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters: 89d5514a6364453581ede69f4179186b
I20240503 08:11:12.189393 25578 tablet_bootstrap.cc:492] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.189499 25575 raft_consensus.cc:348] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.189682 25575 raft_consensus.cc:374] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.189664 25583 raft_consensus.cc:2781] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.189760 25575 raft_consensus.cc:729] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.189785 25583 raft_consensus.cc:482] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.189872 25583 raft_consensus.cc:3037] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.189990 25575 consensus_queue.cc:260] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.190488 25575 ts_tablet_manager.cc:1398] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:12.190757 25575 tablet_bootstrap.cc:492] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.191670 25576 tablet_bootstrap.cc:492] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.191932 25576 ts_tablet_manager.cc:1367] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240503 08:11:12.192350 25575 tablet_bootstrap.cc:654] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.192509 25583 raft_consensus.cc:504] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.192714 25576 raft_consensus.cc:348] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.192899 25576 raft_consensus.cc:374] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.192977 25576 raft_consensus.cc:729] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.189555 25578 ts_tablet_manager.cc:1367] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.014s
I20240503 08:11:12.193648 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81603a53190547b6b5d468d86d374c87" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205"
I20240503 08:11:12.193703 25512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81603a53190547b6b5d468d86d374c87" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.193843 25382 raft_consensus.cc:3037] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:11:12.194229 25188 leader_election.cc:341] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89d5514a6364453581ede69f4179186b (127.24.123.3:44695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.194546 25583 leader_election.cc:288] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 election: Requested vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.195660 25575 tablet_bootstrap.cc:492] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.195797 25575 ts_tablet_manager.cc:1367] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:11:12.196334 25575 raft_consensus.cc:348] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.196492 25575 raft_consensus.cc:374] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.196565 25575 raft_consensus.cc:729] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.196655 25382 raft_consensus.cc:2454] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 1.
I20240503 08:11:12.196805 25575 consensus_queue.cc:260] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.196995 25187 leader_election.cc:302] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters: 89d5514a6364453581ede69f4179186b
I20240503 08:11:12.197294 25575 ts_tablet_manager.cc:1398] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:11:12.197355 25583 raft_consensus.cc:2781] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.196866 25576 consensus_queue.cc:260] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.197557 25575 tablet_bootstrap.cc:492] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.198050 25576 ts_tablet_manager.cc:1398] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:11:12.198315 25576 tablet_bootstrap.cc:492] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.199281 25575 tablet_bootstrap.cc:654] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.199797 25576 tablet_bootstrap.cc:654] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
W20240503 08:11:12.200309 25298 tablet.cc:2369] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 9250c2e966374fc8b65d634d97d41c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:12.201041 25585 raft_consensus.cc:482] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.201186 25585 raft_consensus.cc:504] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.201809 25585 leader_election.cc:288] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:12.202205 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea4abd02e24b98935c3e614c2b7de8" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.202358 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea4abd02e24b98935c3e614c2b7de8" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.202540 25382 raft_consensus.cc:2454] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 0.
I20240503 08:11:12.202929 25447 leader_election.cc:302] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
W20240503 08:11:12.203119 25446 leader_election.cc:341] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.203219 25585 raft_consensus.cc:2781] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.203351 25585 raft_consensus.cc:482] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.203472 25585 raft_consensus.cc:3037] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.206017 25585 raft_consensus.cc:504] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.207024 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea4abd02e24b98935c3e614c2b7de8" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205"
I20240503 08:11:12.207196 25382 raft_consensus.cc:3037] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.207406 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea4abd02e24b98935c3e614c2b7de8" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
W20240503 08:11:12.207912 25446 leader_election.cc:341] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.209252 25578 raft_consensus.cc:348] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.209457 25578 raft_consensus.cc:374] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.209527 25578 raft_consensus.cc:729] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.209798 25382 raft_consensus.cc:2454] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 1.
I20240503 08:11:12.210146 25447 leader_election.cc:302] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters: 9250c2e966374fc8b65d634d97d41c11
I20240503 08:11:12.210281 25578 consensus_queue.cc:260] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.210474 25595 raft_consensus.cc:2781] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.211845 25585 leader_election.cc:288] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:12.214735 25583 raft_consensus.cc:686] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [term 1 LEADER]: Becoming Leader. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Running, Role: LEADER
I20240503 08:11:12.215404 25583 consensus_queue.cc:237] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.221171 25578 ts_tablet_manager.cc:1398] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.028s user 0.004s sys 0.001s
I20240503 08:11:12.221381 25575 tablet_bootstrap.cc:492] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.221525 25578 tablet_bootstrap.cc:492] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.221549 25575 ts_tablet_manager.cc:1367] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.011s
I20240503 08:11:12.222965 25575 raft_consensus.cc:348] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.223166 25575 raft_consensus.cc:374] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.223253 25575 raft_consensus.cc:729] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.223330 25578 tablet_bootstrap.cc:654] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.223500 25575 consensus_queue.cc:260] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.224025 25575 ts_tablet_manager.cc:1398] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:12.224608 25575 tablet_bootstrap.cc:492] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.226109 25575 tablet_bootstrap.cc:654] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.226369 25111 catalog_manager.cc:5484] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "9250c2e966374fc8b65d634d97d41c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.231118 25578 tablet_bootstrap.cc:492] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.231268 25578 ts_tablet_manager.cc:1367] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240503 08:11:12.232043 25578 raft_consensus.cc:348] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.232157 25595 raft_consensus.cc:686] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [term 1 LEADER]: Becoming Leader. State: Replica: 89d5514a6364453581ede69f4179186b, State: Running, Role: LEADER
I20240503 08:11:12.232240 25578 raft_consensus.cc:374] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.232334 25578 raft_consensus.cc:729] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.232594 25578 consensus_queue.cc:260] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.232527 25595 consensus_queue.cc:237] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.233227 25578 ts_tablet_manager.cc:1398] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:12.233496 25578 tablet_bootstrap.cc:492] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.234014 25575 tablet_bootstrap.cc:492] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.234169 25575 ts_tablet_manager.cc:1367] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 08:11:12.235368 25575 raft_consensus.cc:348] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.235602 25575 raft_consensus.cc:374] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.235674 25575 raft_consensus.cc:729] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.235761 25578 tablet_bootstrap.cc:654] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.235927 25575 consensus_queue.cc:260] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.236521 25575 ts_tablet_manager.cc:1398] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:12.236828 25575 tablet_bootstrap.cc:492] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.238299 25575 tablet_bootstrap.cc:654] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.239310 25110 catalog_manager.cc:5484] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b reported cstate change: term changed from 0 to 1, leader changed from <none> to 89d5514a6364453581ede69f4179186b (127.24.123.3). New cstate: current_term: 1 leader_uuid: "89d5514a6364453581ede69f4179186b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:12.239924 25578 tablet_bootstrap.cc:492] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.240057 25578 ts_tablet_manager.cc:1367] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240503 08:11:12.240139 25576 tablet_bootstrap.cc:492] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.240276 25576 ts_tablet_manager.cc:1367] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.000s
I20240503 08:11:12.241019 25576 raft_consensus.cc:348] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.241223 25576 raft_consensus.cc:374] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.241307 25576 raft_consensus.cc:729] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.241266 25578 raft_consensus.cc:348] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.241464 25578 raft_consensus.cc:374] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.241530 25578 raft_consensus.cc:729] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.241551 25576 consensus_queue.cc:260] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.241744 25578 consensus_queue.cc:260] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.242192 25578 ts_tablet_manager.cc:1398] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:11:12.242192 25576 ts_tablet_manager.cc:1398] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:12.242450 25576 tablet_bootstrap.cc:492] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.243942 25576 tablet_bootstrap.cc:654] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.244308 25578 tablet_bootstrap.cc:492] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.245772 25578 tablet_bootstrap.cc:654] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.248915 25575 tablet_bootstrap.cc:492] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.249047 25575 ts_tablet_manager.cc:1367] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240503 08:11:12.249608 25575 raft_consensus.cc:348] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.249755 25575 raft_consensus.cc:374] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.249835 25575 raft_consensus.cc:729] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.250031 25575 consensus_queue.cc:260] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.250330 25583 raft_consensus.cc:482] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.250558 25575 ts_tablet_manager.cc:1398] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:11:12.250499 25583 raft_consensus.cc:504] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.250811 25575 tablet_bootstrap.cc:492] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.251255 25583 leader_election.cc:288] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89d5514a6364453581ede69f4179186b (127.24.123.3:44695), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:12.252478 25512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a82bdbebae44489ec337f79c21507e" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.252795 25575 tablet_bootstrap.cc:654] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.252749 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a82bdbebae44489ec337f79c21507e" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
W20240503 08:11:12.253567 25187 leader_election.cc:341] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:11:12.253932 25188 leader_election.cc:341] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89d5514a6364453581ede69f4179186b (127.24.123.3:44695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.254060 25188 leader_election.cc:302] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11; no voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205
I20240503 08:11:12.254578 25583 raft_consensus.cc:2726] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:12.255859 25578 tablet_bootstrap.cc:492] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.255998 25578 ts_tablet_manager.cc:1367] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20240503 08:11:12.256124 25576 tablet_bootstrap.cc:492] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.256269 25576 ts_tablet_manager.cc:1367] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 08:11:12.256551 25578 raft_consensus.cc:348] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.256685 25578 raft_consensus.cc:374] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.256744 25578 raft_consensus.cc:729] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.256837 25575 tablet_bootstrap.cc:492] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.256913 25578 consensus_queue.cc:260] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.256997 25576 raft_consensus.cc:348] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.257189 25575 ts_tablet_manager.cc:1367] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20240503 08:11:12.257393 25578 ts_tablet_manager.cc:1398] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:12.257465 25576 raft_consensus.cc:374] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.257582 25576 raft_consensus.cc:729] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.257656 25578 tablet_bootstrap.cc:492] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.257800 25576 consensus_queue.cc:260] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.258160 25575 raft_consensus.cc:348] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.258406 25575 raft_consensus.cc:374] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.258517 25575 raft_consensus.cc:729] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.258803 25575 consensus_queue.cc:260] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.259150 25578 tablet_bootstrap.cc:654] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.259390 25576 ts_tablet_manager.cc:1398] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:11:12.259721 25576 tablet_bootstrap.cc:492] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.261796 25576 tablet_bootstrap.cc:654] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.261996 25578 tablet_bootstrap.cc:492] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.262116 25578 ts_tablet_manager.cc:1367] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:11:12.262717 25575 ts_tablet_manager.cc:1398] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:11:12.263043 25575 tablet_bootstrap.cc:492] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.263206 25578 raft_consensus.cc:348] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.263409 25578 raft_consensus.cc:374] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.263490 25578 raft_consensus.cc:729] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.263629 25576 tablet_bootstrap.cc:492] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.263769 25576 ts_tablet_manager.cc:1367] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:11:12.263715 25578 consensus_queue.cc:260] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.264320 25578 ts_tablet_manager.cc:1398] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:12.264530 25575 tablet_bootstrap.cc:654] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.264602 25578 tablet_bootstrap.cc:492] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.264983 25576 raft_consensus.cc:348] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.265201 25576 raft_consensus.cc:374] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.265280 25576 raft_consensus.cc:729] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.265635 25576 consensus_queue.cc:260] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.266098 25576 ts_tablet_manager.cc:1398] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:12.266544 25578 tablet_bootstrap.cc:654] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.266790 25576 tablet_bootstrap.cc:492] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.269378 25576 tablet_bootstrap.cc:654] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.271667 25578 tablet_bootstrap.cc:492] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.271812 25578 ts_tablet_manager.cc:1367] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:11:12.272562 25578 raft_consensus.cc:348] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.272763 25578 raft_consensus.cc:374] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.272854 25578 raft_consensus.cc:729] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.273072 25578 consensus_queue.cc:260] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.273646 25578 ts_tablet_manager.cc:1398] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:12.273923 25578 tablet_bootstrap.cc:492] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.275578 25578 tablet_bootstrap.cc:654] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.276728 25576 tablet_bootstrap.cc:492] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.276861 25576 ts_tablet_manager.cc:1367] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 08:11:12.277372 25578 tablet_bootstrap.cc:492] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.277518 25578 ts_tablet_manager.cc:1367] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:11:12.277586 25575 tablet_bootstrap.cc:492] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.277593 25576 raft_consensus.cc:348] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.277730 25575 ts_tablet_manager.cc:1367] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240503 08:11:12.277784 25576 raft_consensus.cc:374] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.277868 25576 raft_consensus.cc:729] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.278131 25576 consensus_queue.cc:260] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.278178 25578 raft_consensus.cc:348] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.278383 25578 raft_consensus.cc:374] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.278452 25578 raft_consensus.cc:729] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.278394 25575 raft_consensus.cc:348] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.278586 25575 raft_consensus.cc:374] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.278666 25575 raft_consensus.cc:729] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.278656 25578 consensus_queue.cc:260] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.278895 25576 ts_tablet_manager.cc:1398] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:12.278864 25575 consensus_queue.cc:260] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.279145 25576 tablet_bootstrap.cc:492] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.279435 25575 ts_tablet_manager.cc:1398] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:12.279726 25575 tablet_bootstrap.cc:492] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.279997 25578 ts_tablet_manager.cc:1398] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:12.280246 25578 tablet_bootstrap.cc:492] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.280607 25576 tablet_bootstrap.cc:654] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.281162 25575 tablet_bootstrap.cc:654] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.281622 25578 tablet_bootstrap.cc:654] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.285153 25579 raft_consensus.cc:482] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.285322 25579 raft_consensus.cc:504] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.287406 25579 leader_election.cc:288] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.289748 25595 raft_consensus.cc:482] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.289923 25595 raft_consensus.cc:504] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.290570 25595 leader_election.cc:288] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:12.291388 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a82bdbebae44489ec337f79c21507e" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.291654 25251 raft_consensus.cc:2454] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 0.
I20240503 08:11:12.292001 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a82bdbebae44489ec337f79c21507e" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.292222 25382 raft_consensus.cc:2454] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 0.
I20240503 08:11:12.292692 25446 leader_election.cc:302] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, 9250c2e966374fc8b65d634d97d41c11; no voters:
I20240503 08:11:12.293471 25595 raft_consensus.cc:2781] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.293586 25595 raft_consensus.cc:482] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.293696 25595 raft_consensus.cc:3037] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.296907 25595 raft_consensus.cc:504] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.298249 25583 raft_consensus.cc:482] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.298477 25583 raft_consensus.cc:504] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.299237 25583 leader_election.cc:288] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.299705 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a82bdbebae44489ec337f79c21507e" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.299942 25251 raft_consensus.cc:3037] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.303179 25251 raft_consensus.cc:2454] T 48a82bdbebae44489ec337f79c21507e P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 1.
I20240503 08:11:12.303929 25446 leader_election.cc:302] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, 9250c2e966374fc8b65d634d97d41c11; no voters:
I20240503 08:11:12.304270 25609 raft_consensus.cc:2781] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.304435 25609 raft_consensus.cc:686] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [term 1 LEADER]: Becoming Leader. State: Replica: 89d5514a6364453581ede69f4179186b, State: Running, Role: LEADER
I20240503 08:11:12.304574 25512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fdef04645494e8569400bb5ae66ce" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.304770 25609 consensus_queue.cc:237] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } }
I20240503 08:11:12.305454 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fdef04645494e8569400bb5ae66ce" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
W20240503 08:11:12.306066 25187 leader_election.cc:341] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:11:12.306349 25188 leader_election.cc:341] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89d5514a6364453581ede69f4179186b (127.24.123.3:44695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.306500 25188 leader_election.cc:302] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11; no voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205
I20240503 08:11:12.306980 25583 raft_consensus.cc:2726] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:12.307312 25579 raft_consensus.cc:482] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.307525 25579 raft_consensus.cc:504] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.308249 25579 leader_election.cc:288] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.309145 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a82bdbebae44489ec337f79c21507e" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205"
I20240503 08:11:12.309419 25382 raft_consensus.cc:3037] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.312548 25382 raft_consensus.cc:2454] T 48a82bdbebae44489ec337f79c21507e P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 1.
I20240503 08:11:12.313231 25595 leader_election.cc:288] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:12.319581 25111 catalog_manager.cc:5484] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b reported cstate change: term changed from 0 to 1, leader changed from <none> to 89d5514a6364453581ede69f4179186b (127.24.123.3). New cstate: current_term: 1 leader_uuid: "89d5514a6364453581ede69f4179186b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.321924 25576 tablet_bootstrap.cc:492] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.321926 25575 tablet_bootstrap.cc:492] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.322093 25576 ts_tablet_manager.cc:1367] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.006s
I20240503 08:11:12.322193 25575 ts_tablet_manager.cc:1367] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.025s
I20240503 08:11:12.322908 25576 raft_consensus.cc:348] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.323134 25576 raft_consensus.cc:374] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.323207 25576 raft_consensus.cc:729] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.323012 25575 raft_consensus.cc:348] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.323448 25575 raft_consensus.cc:374] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.323472 25576 consensus_queue.cc:260] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.323580 25575 raft_consensus.cc:729] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.323972 25576 ts_tablet_manager.cc:1398] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:12.324328 25576 tablet_bootstrap.cc:492] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.324246 25575 consensus_queue.cc:260] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.324962 25575 ts_tablet_manager.cc:1398] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:12.325353 25575 tablet_bootstrap.cc:492] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.325867 25576 tablet_bootstrap.cc:654] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.326900 25575 tablet_bootstrap.cc:654] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.327925 25578 tablet_bootstrap.cc:492] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.328969 25578 ts_tablet_manager.cc:1367] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.027s
I20240503 08:11:12.329835 25578 raft_consensus.cc:348] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.330090 25578 raft_consensus.cc:374] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.330176 25578 raft_consensus.cc:729] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.330399 25578 consensus_queue.cc:260] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.331048 25578 ts_tablet_manager.cc:1398] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:12.331396 25578 tablet_bootstrap.cc:492] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.332254 25512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5437b33ba439f86ded3348da6159a" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.332518 25512 raft_consensus.cc:2454] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.332923 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fda691c4a54496b216ca52582b3aa2" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.333240 25511 raft_consensus.cc:2454] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.333629 25578 tablet_bootstrap.cc:654] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.333845 25318 leader_election.cc:302] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.334323 25318 leader_election.cc:302] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.334718 25579 raft_consensus.cc:2781] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.334810 25579 raft_consensus.cc:482] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.334961 25579 raft_consensus.cc:3037] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.335561 25576 tablet_bootstrap.cc:492] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.335692 25576 ts_tablet_manager.cc:1367] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.009s
I20240503 08:11:12.336417 25575 tablet_bootstrap.cc:492] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.336434 25576 raft_consensus.cc:348] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.336572 25575 ts_tablet_manager.cc:1367] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240503 08:11:12.336624 25576 raft_consensus.cc:374] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.336701 25576 raft_consensus.cc:729] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.336901 25576 consensus_queue.cc:260] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.337592 25576 ts_tablet_manager.cc:1398] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:12.337860 25576 tablet_bootstrap.cc:492] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.337891 25579 raft_consensus.cc:504] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.338711 25579 leader_election.cc:288] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
W20240503 08:11:12.339172 25558 tablet.cc:2369] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:12.339795 25576 tablet_bootstrap.cc:654] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.341012 25609 raft_consensus.cc:482] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.341188 25579 raft_consensus.cc:2781] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.341171 25609 raft_consensus.cc:504] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.341514 25579 raft_consensus.cc:482] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.341825 25609 leader_election.cc:288] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:12.342383 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5437b33ba439f86ded3348da6159a" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.342495 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f0fba475d84890a0b1496fdbe0ba28" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.342586 25511 raft_consensus.cc:3037] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.343034 25583 raft_consensus.cc:482] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 08:11:12.343111 25446 leader_election.cc:341] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.343202 25583 raft_consensus.cc:504] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.343871 25583 leader_election.cc:288] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.343928 25578 tablet_bootstrap.cc:492] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.344115 25578 ts_tablet_manager.cc:1367] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20240503 08:11:12.344869 25578 raft_consensus.cc:348] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.345113 25578 raft_consensus.cc:374] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.345186 25578 raft_consensus.cc:729] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.345305 25511 raft_consensus.cc:2454] T f4d5437b33ba439f86ded3348da6159a P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.345414 25578 consensus_queue.cc:260] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.343227 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f0fba475d84890a0b1496fdbe0ba28" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.345790 25579 raft_consensus.cc:3037] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.345840 25318 leader_election.cc:302] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.345873 25382 raft_consensus.cc:2454] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 0.
I20240503 08:11:12.346014 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a899d61a8b481eb8f02bd4f1997a05" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.346195 25511 raft_consensus.cc:2454] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 0.
I20240503 08:11:12.346279 25447 leader_election.cc:302] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters: 9250c2e966374fc8b65d634d97d41c11
I20240503 08:11:12.346542 25188 leader_election.cc:302] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, 9250c2e966374fc8b65d634d97d41c11; no voters:
I20240503 08:11:12.346661 25609 raft_consensus.cc:2781] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.346733 25609 raft_consensus.cc:482] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.346820 25609 raft_consensus.cc:3037] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.346817 25583 raft_consensus.cc:2781] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.346904 25583 raft_consensus.cc:482] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.346992 25583 raft_consensus.cc:3037] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.348372 25634 raft_consensus.cc:2781] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.348351 25579 raft_consensus.cc:504] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.349041 25579 leader_election.cc:288] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.349287 25575 raft_consensus.cc:348] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.349313 25609 raft_consensus.cc:504] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.349503 25575 raft_consensus.cc:374] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.349421 25583 raft_consensus.cc:504] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.349932 25609 leader_election.cc:288] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:12.350109 25575 raft_consensus.cc:729] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.350075 25512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fda691c4a54496b216ca52582b3aa2" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.344488 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a899d61a8b481eb8f02bd4f1997a05" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.350171 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f0fba475d84890a0b1496fdbe0ba28" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.350240 25512 raft_consensus.cc:3037] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.350292 25381 raft_consensus.cc:2454] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 0.
I20240503 08:11:12.350390 25575 consensus_queue.cc:260] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
W20240503 08:11:12.350641 25446 leader_election.cc:341] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.351065 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f0fba475d84890a0b1496fdbe0ba28" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205"
I20240503 08:11:12.351155 25575 ts_tablet_manager.cc:1398] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.014s user 0.002s sys 0.002s
I20240503 08:11:12.351235 25381 raft_consensus.cc:3037] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.351469 25575 tablet_bootstrap.cc:492] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.351493 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a899d61a8b481eb8f02bd4f1997a05" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.351799 25510 raft_consensus.cc:3037] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.353173 25512 raft_consensus.cc:2454] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.353327 25575 tablet_bootstrap.cc:654] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.354140 25381 raft_consensus.cc:2454] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 1.
I20240503 08:11:12.354317 25318 leader_election.cc:302] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.354424 25510 raft_consensus.cc:2454] T b7a899d61a8b481eb8f02bd4f1997a05 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 1.
I20240503 08:11:12.354753 25447 leader_election.cc:302] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters: 9250c2e966374fc8b65d634d97d41c11
I20240503 08:11:12.354840 25188 leader_election.cc:302] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, 9250c2e966374fc8b65d634d97d41c11; no voters:
I20240503 08:11:12.355139 25609 raft_consensus.cc:2781] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.355129 25608 raft_consensus.cc:2781] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.355273 25608 raft_consensus.cc:686] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [term 1 LEADER]: Becoming Leader. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Running, Role: LEADER
I20240503 08:11:12.355367 25609 raft_consensus.cc:686] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [term 1 LEADER]: Becoming Leader. State: Replica: 89d5514a6364453581ede69f4179186b, State: Running, Role: LEADER
I20240503 08:11:12.355607 25609 consensus_queue.cc:237] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.355589 25608 consensus_queue.cc:237] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.356602 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a899d61a8b481eb8f02bd4f1997a05" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205"
I20240503 08:11:12.356783 25381 raft_consensus.cc:3037] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.357901 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5437b33ba439f86ded3348da6159a" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.357980 25249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fda691c4a54496b216ca52582b3aa2" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.358322 25252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fda691c4a54496b216ca52582b3aa2" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.358772 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d5437b33ba439f86ded3348da6159a" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
W20240503 08:11:12.358801 25316 leader_election.cc:341] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:11:12.359133 25316 leader_election.cc:341] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.359386 25381 raft_consensus.cc:2454] T b7a899d61a8b481eb8f02bd4f1997a05 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 1.
I20240503 08:11:12.359422 25583 leader_election.cc:288] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 election: Requested vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
W20240503 08:11:12.359674 25316 leader_election.cc:341] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:11:12.359968 25316 leader_election.cc:341] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.360193 25579 raft_consensus.cc:482] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.360538 25579 raft_consensus.cc:504] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.361462 25579 leader_election.cc:288] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.361791 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286d705044a24c9b995fa8795df2205a" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.362015 25251 raft_consensus.cc:2454] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.362342 25579 raft_consensus.cc:2781] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.362525 25316 leader_election.cc:302] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.363405 25579 raft_consensus.cc:686] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [term 1 LEADER]: Becoming Leader. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Running, Role: LEADER
I20240503 08:11:12.363942 25579 consensus_queue.cc:237] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.364341 25595 raft_consensus.cc:482] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.364441 25578 ts_tablet_manager.cc:1398] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240503 08:11:12.364513 25595 raft_consensus.cc:504] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.364751 25578 tablet_bootstrap.cc:492] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.365120 25595 leader_election.cc:288] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251)
I20240503 08:11:12.365506 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba38aecd198c4920942d189fbfe49653" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.365712 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba38aecd198c4920942d189fbfe49653" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.365758 25251 raft_consensus.cc:2454] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 0.
I20240503 08:11:12.365883 25381 raft_consensus.cc:2454] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 0.
I20240503 08:11:12.366217 25447 leader_election.cc:302] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.366413 25583 raft_consensus.cc:482] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.366611 25595 raft_consensus.cc:2781] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.366699 25595 raft_consensus.cc:482] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.366796 25595 raft_consensus.cc:3037] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.366611 25583 raft_consensus.cc:504] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.367753 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae79d8226b86428fa40e03c568813658" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.368014 25511 raft_consensus.cc:2454] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 0.
I20240503 08:11:12.368510 25188 leader_election.cc:302] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, 9250c2e966374fc8b65d634d97d41c11; no voters:
I20240503 08:11:12.368772 25605 raft_consensus.cc:2781] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.368845 25605 raft_consensus.cc:482] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.368925 25605 raft_consensus.cc:3037] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.369303 25595 raft_consensus.cc:504] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.369859 25595 leader_election.cc:288] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Requested vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251)
I20240503 08:11:12.370245 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae79d8226b86428fa40e03c568813658" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.370424 25382 raft_consensus.cc:2454] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 0.
I20240503 08:11:12.371511 25605 raft_consensus.cc:504] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.372102 25605 leader_election.cc:288] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 election: Requested vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.372925 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba38aecd198c4920942d189fbfe49653" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205"
I20240503 08:11:12.373031 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba38aecd198c4920942d189fbfe49653" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.373203 25251 raft_consensus.cc:3037] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.373217 25382 raft_consensus.cc:3037] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.375763 25251 raft_consensus.cc:2454] T ba38aecd198c4920942d189fbfe49653 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 1.
I20240503 08:11:12.376260 25446 leader_election.cc:302] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, 9250c2e966374fc8b65d634d97d41c11; no voters:
I20240503 08:11:12.376327 25382 raft_consensus.cc:2454] T ba38aecd198c4920942d189fbfe49653 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89d5514a6364453581ede69f4179186b in term 1.
I20240503 08:11:12.376644 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae79d8226b86428fa40e03c568813658" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.376866 25510 raft_consensus.cc:3037] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.376880 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae79d8226b86428fa40e03c568813658" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205"
I20240503 08:11:12.377126 25381 raft_consensus.cc:3037] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.379388 25510 raft_consensus.cc:2454] T ae79d8226b86428fa40e03c568813658 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 1.
I20240503 08:11:12.379808 25188 leader_election.cc:302] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, 9250c2e966374fc8b65d634d97d41c11; no voters:
I20240503 08:11:12.380065 25381 raft_consensus.cc:2454] T ae79d8226b86428fa40e03c568813658 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 1.
I20240503 08:11:12.380156 25605 raft_consensus.cc:2781] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.380475 25595 raft_consensus.cc:2781] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.380697 25595 raft_consensus.cc:686] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [term 1 LEADER]: Becoming Leader. State: Replica: 89d5514a6364453581ede69f4179186b, State: Running, Role: LEADER
I20240503 08:11:12.380964 25595 consensus_queue.cc:237] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.382002 25605 raft_consensus.cc:686] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [term 1 LEADER]: Becoming Leader. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Running, Role: LEADER
I20240503 08:11:12.382258 25605 consensus_queue.cc:237] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.383587 25583 leader_election.cc:288] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.387146 25579 raft_consensus.cc:2781] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.387264 25605 raft_consensus.cc:482] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.387396 25605 raft_consensus.cc:504] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.388422 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6175bd8e1b7344d086a6ad2d59b50841" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.388711 25511 raft_consensus.cc:2454] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 0.
I20240503 08:11:12.389017 25579 raft_consensus.cc:482] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.388401 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6175bd8e1b7344d086a6ad2d59b50841" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.389114 25188 leader_election.cc:302] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, 9250c2e966374fc8b65d634d97d41c11; no voters:
I20240503 08:11:12.389482 25583 raft_consensus.cc:2781] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.389614 25583 raft_consensus.cc:482] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.389758 25583 raft_consensus.cc:3037] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.389842 25576 tablet_bootstrap.cc:492] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.390002 25576 ts_tablet_manager.cc:1367] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.010s
I20240503 08:11:12.390158 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286d705044a24c9b995fa8795df2205a" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.390475 25511 raft_consensus.cc:2454] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.390761 25576 raft_consensus.cc:348] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.390947 25605 leader_election.cc:288] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.390991 25576 raft_consensus.cc:374] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.391083 25576 raft_consensus.cc:729] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.391288 25576 consensus_queue.cc:260] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.391552 25381 raft_consensus.cc:2454] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 0.
I20240503 08:11:12.391914 25579 raft_consensus.cc:3037] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.392446 25638 raft_consensus.cc:482] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.392639 25638 raft_consensus.cc:504] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.392856 25583 raft_consensus.cc:504] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.393519 25576 ts_tablet_manager.cc:1398] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:12.393841 25576 tablet_bootstrap.cc:492] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.393743 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6175bd8e1b7344d086a6ad2d59b50841" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205"
I20240503 08:11:12.393958 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6175bd8e1b7344d086a6ad2d59b50841" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.394073 25381 raft_consensus.cc:3037] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.394215 25511 raft_consensus.cc:3037] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.395431 25576 tablet_bootstrap.cc:654] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.396968 25511 raft_consensus.cc:2454] T 6175bd8e1b7344d086a6ad2d59b50841 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 1.
I20240503 08:11:12.397405 25188 leader_election.cc:302] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, 9250c2e966374fc8b65d634d97d41c11; no voters:
I20240503 08:11:12.397768 25605 raft_consensus.cc:2781] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.397946 25605 raft_consensus.cc:686] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [term 1 LEADER]: Becoming Leader. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Running, Role: LEADER
I20240503 08:11:12.398244 25605 consensus_queue.cc:237] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.398910 25579 raft_consensus.cc:504] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.400156 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286d705044a24c9b995fa8795df2205a" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.400583 25252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2291b936dd324f0fbaa0ff23b1369447" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.400784 25252 raft_consensus.cc:2454] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.400853 25381 raft_consensus.cc:2454] T 6175bd8e1b7344d086a6ad2d59b50841 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9250c2e966374fc8b65d634d97d41c11 in term 1.
I20240503 08:11:12.401250 25316 leader_election.cc:302] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.400585 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286d705044a24c9b995fa8795df2205a" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.401867 25251 raft_consensus.cc:3037] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.401804 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2291b936dd324f0fbaa0ff23b1369447" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.402026 25576 tablet_bootstrap.cc:492] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.402621 25576 ts_tablet_manager.cc:1367] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
W20240503 08:11:12.402729 25318 leader_election.cc:341] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89d5514a6364453581ede69f4179186b (127.24.123.3:44695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:12.402989 25638 leader_election.cc:288] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.403275 25576 raft_consensus.cc:348] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.403470 25576 raft_consensus.cc:374] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.403561 25576 raft_consensus.cc:729] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.403762 25638 raft_consensus.cc:2781] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.403865 25638 raft_consensus.cc:482] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.403774 25576 consensus_queue.cc:260] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.403947 25638 raft_consensus.cc:3037] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.404321 25575 tablet_bootstrap.cc:492] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.404476 25251 raft_consensus.cc:2454] T 286d705044a24c9b995fa8795df2205a P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.404523 25575 ts_tablet_manager.cc:1367] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.053s user 0.025s sys 0.020s
I20240503 08:11:12.404991 25316 leader_election.cc:302] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.400367 25511 raft_consensus.cc:3037] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.405222 25575 raft_consensus.cc:348] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.405442 25575 raft_consensus.cc:374] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.405525 25575 raft_consensus.cc:729] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.405843 25576 ts_tablet_manager.cc:1398] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:12.405769 25575 consensus_queue.cc:260] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.406100 25576 tablet_bootstrap.cc:492] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.407024 25579 leader_election.cc:288] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.407192 25579 raft_consensus.cc:2781] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.407332 25579 raft_consensus.cc:686] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [term 1 LEADER]: Becoming Leader. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Running, Role: LEADER
I20240503 08:11:12.407656 25579 consensus_queue.cc:237] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.409173 25511 raft_consensus.cc:2454] T 286d705044a24c9b995fa8795df2205a P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.409516 25576 tablet_bootstrap.cc:654] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.409582 25638 raft_consensus.cc:504] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.410576 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2291b936dd324f0fbaa0ff23b1369447" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.410836 25251 raft_consensus.cc:3037] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.411262 25583 leader_election.cc:288] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 election: Requested vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.413672 25575 ts_tablet_manager.cc:1398] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20240503 08:11:12.413957 25575 tablet_bootstrap.cc:492] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205: Bootstrap starting.
I20240503 08:11:12.413941 25251 raft_consensus.cc:2454] T 2291b936dd324f0fbaa0ff23b1369447 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.414378 25316 leader_election.cc:302] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.415784 25575 tablet_bootstrap.cc:654] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.416828 25647 raft_consensus.cc:2781] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.416873 25578 tablet_bootstrap.cc:654] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.416958 25647 raft_consensus.cc:686] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [term 1 LEADER]: Becoming Leader. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Running, Role: LEADER
I20240503 08:11:12.417784 25647 consensus_queue.cc:237] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.418618 25638 leader_election.cc:288] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.419626 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2291b936dd324f0fbaa0ff23b1369447" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.419900 25511 raft_consensus.cc:3037] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.422233 25576 tablet_bootstrap.cc:492] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.422403 25576 ts_tablet_manager.cc:1367] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240503 08:11:12.422850 25511 raft_consensus.cc:2454] T 2291b936dd324f0fbaa0ff23b1369447 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.423131 25576 raft_consensus.cc:348] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.423354 25576 raft_consensus.cc:374] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.423434 25576 raft_consensus.cc:729] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.424919 25576 consensus_queue.cc:260] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.425539 25576 ts_tablet_manager.cc:1398] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:12.426134 25575 tablet_bootstrap.cc:492] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205: No bootstrap required, opened a new log
I20240503 08:11:12.426272 25575 ts_tablet_manager.cc:1367] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240503 08:11:12.426373 25110 catalog_manager.cc:5484] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "9250c2e966374fc8b65d634d97d41c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.426779 25111 catalog_manager.cc:5484] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b reported cstate change: term changed from 0 to 1, leader changed from <none> to 89d5514a6364453581ede69f4179186b (127.24.123.3). New cstate: current_term: 1 leader_uuid: "89d5514a6364453581ede69f4179186b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:12.427064 25575 raft_consensus.cc:348] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.427273 25575 raft_consensus.cc:374] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.427343 25575 raft_consensus.cc:729] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.427572 25575 consensus_queue.cc:260] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.427703 25576 tablet_bootstrap.cc:492] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b: Bootstrap starting.
I20240503 08:11:12.428139 25575 ts_tablet_manager.cc:1398] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:12.428972 25576 tablet_bootstrap.cc:654] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.431773 25578 tablet_bootstrap.cc:492] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.431913 25578 ts_tablet_manager.cc:1367] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.001s
I20240503 08:11:12.432657 25578 raft_consensus.cc:348] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.432857 25578 raft_consensus.cc:374] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.432947 25578 raft_consensus.cc:729] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.433169 25578 consensus_queue.cc:260] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.433723 25578 ts_tablet_manager.cc:1398] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:12.433959 25578 tablet_bootstrap.cc:492] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.435289 25578 tablet_bootstrap.cc:654] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.437391 25634 raft_consensus.cc:686] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [term 1 LEADER]: Becoming Leader. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Running, Role: LEADER
I20240503 08:11:12.437760 25634 consensus_queue.cc:237] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.439859 25576 tablet_bootstrap.cc:492] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b: No bootstrap required, opened a new log
I20240503 08:11:12.441067 25111 catalog_manager.cc:5484] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b reported cstate change: term changed from 0 to 1, leader changed from <none> to 89d5514a6364453581ede69f4179186b (127.24.123.3). New cstate: current_term: 1 leader_uuid: "89d5514a6364453581ede69f4179186b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:12.443105 25110 catalog_manager.cc:5484] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "9250c2e966374fc8b65d634d97d41c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.443670 25110 catalog_manager.cc:5484] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "9250c2e966374fc8b65d634d97d41c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.444972 25576 ts_tablet_manager.cc:1367] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20240503 08:11:12.445760 25576 raft_consensus.cc:348] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.445992 25576 raft_consensus.cc:374] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.446338 25111 catalog_manager.cc:5484] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 reported cstate change: term changed from 0 to 1, leader changed from <none> to baf64abf9ec642e0acb49eb700e54205 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "baf64abf9ec642e0acb49eb700e54205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.446645 25576 raft_consensus.cc:729] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5514a6364453581ede69f4179186b, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.447121 25578 tablet_bootstrap.cc:492] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.446995 25576 consensus_queue.cc:260] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.447280 25578 ts_tablet_manager.cc:1367] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240503 08:11:12.447695 25576 ts_tablet_manager.cc:1398] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:12.447974 25578 raft_consensus.cc:348] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.448211 25578 raft_consensus.cc:374] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.448302 25578 raft_consensus.cc:729] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.448527 25578 consensus_queue.cc:260] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.448776 25634 raft_consensus.cc:482] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.448935 25634 raft_consensus.cc:504] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.449894 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe51bafa1a24fe0bd34f0d2333ab9b5" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.450074 25251 raft_consensus.cc:2454] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.450506 25316 leader_election.cc:302] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.450603 25578 ts_tablet_manager.cc:1398] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:11:12.450862 25578 tablet_bootstrap.cc:492] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11: Bootstrap starting.
I20240503 08:11:12.451097 25636 raft_consensus.cc:2781] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.451239 25636 raft_consensus.cc:482] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.451468 25636 raft_consensus.cc:3037] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.453013 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe51bafa1a24fe0bd34f0d2333ab9b5" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.453171 25578 tablet_bootstrap.cc:654] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:12.453217 25511 raft_consensus.cc:2454] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.453650 25647 raft_consensus.cc:482] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.453671 25634 leader_election.cc:288] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.453811 25647 raft_consensus.cc:504] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.454574 25647 leader_election.cc:288] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.454555 25110 catalog_manager.cc:5484] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 reported cstate change: term changed from 0 to 1, leader changed from <none> to baf64abf9ec642e0acb49eb700e54205 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "baf64abf9ec642e0acb49eb700e54205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.455016 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fdef04645494e8569400bb5ae66ce" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.455114 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fdef04645494e8569400bb5ae66ce" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.455217 25251 raft_consensus.cc:2454] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.455292 25511 raft_consensus.cc:2454] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.455135 25110 catalog_manager.cc:5484] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 reported cstate change: term changed from 0 to 1, leader changed from <none> to baf64abf9ec642e0acb49eb700e54205 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "baf64abf9ec642e0acb49eb700e54205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.455659 25316 leader_election.cc:302] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.455667 25110 catalog_manager.cc:5484] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 reported cstate change: term changed from 0 to 1, leader changed from <none> to baf64abf9ec642e0acb49eb700e54205 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "baf64abf9ec642e0acb49eb700e54205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.456193 25647 raft_consensus.cc:2781] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.456288 25647 raft_consensus.cc:482] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.456383 25647 raft_consensus.cc:3037] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.457093 25636 raft_consensus.cc:504] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.457747 25636 leader_election.cc:288] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.458104 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe51bafa1a24fe0bd34f0d2333ab9b5" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.458226 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe51bafa1a24fe0bd34f0d2333ab9b5" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.458288 25251 raft_consensus.cc:3037] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.458384 25511 raft_consensus.cc:3037] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.458822 25578 tablet_bootstrap.cc:492] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11: No bootstrap required, opened a new log
I20240503 08:11:12.458973 25578 ts_tablet_manager.cc:1367] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 08:11:12.459762 25578 raft_consensus.cc:348] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.459954 25578 raft_consensus.cc:374] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:12.460037 25578 raft_consensus.cc:729] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250c2e966374fc8b65d634d97d41c11, State: Initialized, Role: FOLLOWER
I20240503 08:11:12.460335 25578 consensus_queue.cc:260] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.460717 25647 raft_consensus.cc:504] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.460914 25578 ts_tablet_manager.cc:1398] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:12.460984 25511 raft_consensus.cc:2454] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.461388 25318 leader_election.cc:302] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.461459 25647 leader_election.cc:288] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.461623 25252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fdef04645494e8569400bb5ae66ce" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.461810 25252 raft_consensus.cc:3037] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.461952 25647 raft_consensus.cc:2781] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.462097 25647 raft_consensus.cc:686] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [term 1 LEADER]: Becoming Leader. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Running, Role: LEADER
I20240503 08:11:12.462262 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fdef04645494e8569400bb5ae66ce" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.462430 25510 raft_consensus.cc:3037] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.462355 25647 consensus_queue.cc:237] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.464200 25252 raft_consensus.cc:2454] T cc7fdef04645494e8569400bb5ae66ce P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.464435 25251 raft_consensus.cc:2454] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.464598 25316 leader_election.cc:302] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.465023 25510 raft_consensus.cc:2454] T cc7fdef04645494e8569400bb5ae66ce P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.465148 25636 raft_consensus.cc:2781] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.465338 25636 raft_consensus.cc:686] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [term 1 LEADER]: Becoming Leader. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Running, Role: LEADER
I20240503 08:11:12.465695 25636 consensus_queue.cc:237] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.466544 25110 catalog_manager.cc:5484] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 reported cstate change: term changed from 0 to 1, leader changed from <none> to baf64abf9ec642e0acb49eb700e54205 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "baf64abf9ec642e0acb49eb700e54205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.470270 25110 catalog_manager.cc:5484] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 reported cstate change: term changed from 0 to 1, leader changed from <none> to baf64abf9ec642e0acb49eb700e54205 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "baf64abf9ec642e0acb49eb700e54205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.473167 25609 raft_consensus.cc:482] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.473292 25609 raft_consensus.cc:504] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.473858 25609 leader_election.cc:288] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849)
I20240503 08:11:12.474247 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81603a53190547b6b5d468d86d374c87" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.474301 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81603a53190547b6b5d468d86d374c87" candidate_uuid: "89d5514a6364453581ede69f4179186b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.474519 25381 raft_consensus.cc:2379] T 81603a53190547b6b5d468d86d374c87 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89d5514a6364453581ede69f4179186b in current term 1: Already voted for candidate 9250c2e966374fc8b65d634d97d41c11 in this term.
I20240503 08:11:12.475251 25446 leader_election.cc:302] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89d5514a6364453581ede69f4179186b; no voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205
I20240503 08:11:12.475512 25609 raft_consensus.cc:3037] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.477269 25609 raft_consensus.cc:2726] T 81603a53190547b6b5d468d86d374c87 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:12.506583 25605 raft_consensus.cc:482] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.506781 25605 raft_consensus.cc:504] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.507473 25605 leader_election.cc:288] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.507977 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea4abd02e24b98935c3e614c2b7de8" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.508013 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea4abd02e24b98935c3e614c2b7de8" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.508245 25381 raft_consensus.cc:2379] T f7ea4abd02e24b98935c3e614c2b7de8 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9250c2e966374fc8b65d634d97d41c11 in current term 1: Already voted for candidate 89d5514a6364453581ede69f4179186b in this term.
I20240503 08:11:12.508765 25187 leader_election.cc:302] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11; no voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205
I20240503 08:11:12.509032 25605 raft_consensus.cc:3037] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.511013 25605 raft_consensus.cc:2726] T f7ea4abd02e24b98935c3e614c2b7de8 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:12.527832 25636 raft_consensus.cc:482] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.527980 25636 raft_consensus.cc:504] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.528558 25636 leader_election.cc:288] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.529021 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06530128c24242828d4a602f75b6bfe7" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.529054 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06530128c24242828d4a602f75b6bfe7" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11" is_pre_election: true
I20240503 08:11:12.529219 25510 raft_consensus.cc:2454] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.529248 25251 raft_consensus.cc:2454] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 0.
I20240503 08:11:12.529628 25318 leader_election.cc:302] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.529873 25636 raft_consensus.cc:2781] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:12.529992 25636 raft_consensus.cc:482] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:12.530086 25636 raft_consensus.cc:3037] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.532137 25636 raft_consensus.cc:504] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.532729 25636 leader_election.cc:288] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Requested vote from peers 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.533120 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06530128c24242828d4a602f75b6bfe7" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b"
I20240503 08:11:12.533114 25251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06530128c24242828d4a602f75b6bfe7" candidate_uuid: "baf64abf9ec642e0acb49eb700e54205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250c2e966374fc8b65d634d97d41c11"
I20240503 08:11:12.533265 25251 raft_consensus.cc:3037] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.533291 25510 raft_consensus.cc:3037] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.535223 25251 raft_consensus.cc:2454] T 06530128c24242828d4a602f75b6bfe7 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.535534 25510 raft_consensus.cc:2454] T 06530128c24242828d4a602f75b6bfe7 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf64abf9ec642e0acb49eb700e54205 in term 1.
I20240503 08:11:12.535571 25316 leader_election.cc:302] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11, baf64abf9ec642e0acb49eb700e54205; no voters:
I20240503 08:11:12.535812 25636 raft_consensus.cc:2781] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:12.535936 25636 raft_consensus.cc:686] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [term 1 LEADER]: Becoming Leader. State: Replica: baf64abf9ec642e0acb49eb700e54205, State: Running, Role: LEADER
I20240503 08:11:12.536257 25636 consensus_queue.cc:237] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.538532 25110 catalog_manager.cc:5484] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 reported cstate change: term changed from 0 to 1, leader changed from <none> to baf64abf9ec642e0acb49eb700e54205 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "baf64abf9ec642e0acb49eb700e54205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:12.595434 25609 consensus_queue.cc:1035] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.601019 25585 consensus_queue.cc:1035] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.603730 25605 consensus_queue.cc:1035] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.608403 25583 consensus_queue.cc:1035] T 81603a53190547b6b5d468d86d374c87 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.689364 25585 consensus_queue.cc:1035] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.694541 25583 raft_consensus.cc:482] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.694696 25583 raft_consensus.cc:504] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.695333 25583 leader_election.cc:288] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.696058 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fda691c4a54496b216ca52582b3aa2" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.696079 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fda691c4a54496b216ca52582b3aa2" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.696312 25510 raft_consensus.cc:2379] T 92fda691c4a54496b216ca52582b3aa2 P 89d5514a6364453581ede69f4179186b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9250c2e966374fc8b65d634d97d41c11 in current term 1: Already voted for candidate baf64abf9ec642e0acb49eb700e54205 in this term.
I20240503 08:11:12.696851 25187 leader_election.cc:302] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11; no voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205
I20240503 08:11:12.697274 25583 raft_consensus.cc:3037] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.697643 25609 consensus_queue.cc:1035] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.699910 25583 raft_consensus.cc:2726] T 92fda691c4a54496b216ca52582b3aa2 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:12.748558 25609 consensus_queue.cc:1035] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.753542 25609 consensus_queue.cc:1035] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.760138 25583 raft_consensus.cc:482] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:12.760310 25583 raft_consensus.cc:504] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 } } peers { permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 } } peers { permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 } }
I20240503 08:11:12.760864 25583 leader_election.cc:288] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf64abf9ec642e0acb49eb700e54205 (127.24.123.2:32849), 89d5514a6364453581ede69f4179186b (127.24.123.3:44695)
I20240503 08:11:12.761257 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f0fba475d84890a0b1496fdbe0ba28" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf64abf9ec642e0acb49eb700e54205" is_pre_election: true
I20240503 08:11:12.761468 25381 raft_consensus.cc:2379] T 53f0fba475d84890a0b1496fdbe0ba28 P baf64abf9ec642e0acb49eb700e54205 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9250c2e966374fc8b65d634d97d41c11 in current term 1: Already voted for candidate 89d5514a6364453581ede69f4179186b in this term.
I20240503 08:11:12.761375 25510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f0fba475d84890a0b1496fdbe0ba28" candidate_uuid: "9250c2e966374fc8b65d634d97d41c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d5514a6364453581ede69f4179186b" is_pre_election: true
I20240503 08:11:12.761973 25188 leader_election.cc:302] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250c2e966374fc8b65d634d97d41c11; no voters: 89d5514a6364453581ede69f4179186b, baf64abf9ec642e0acb49eb700e54205
I20240503 08:11:12.762212 25583 raft_consensus.cc:3037] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.764801 25583 raft_consensus.cc:2726] T 53f0fba475d84890a0b1496fdbe0ba28 P 9250c2e966374fc8b65d634d97d41c11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:12.792915 25636 consensus_queue.cc:1035] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.820982 25638 consensus_queue.cc:1035] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.831867 25638 consensus_queue.cc:1035] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.836285 25636 consensus_queue.cc:1035] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.836755 25585 consensus_queue.cc:1035] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.844424 25595 consensus_queue.cc:1035] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.852157 25585 consensus_queue.cc:1035] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.856190 25681 consensus_queue.cc:1035] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.867980 25583 consensus_queue.cc:1035] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.873203 25641 consensus_queue.cc:1035] T ae79d8226b86428fa40e03c568813658 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.888016 25636 consensus_queue.cc:1035] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.891414 25252 raft_consensus.cc:3037] T f4d5437b33ba439f86ded3348da6159a P 9250c2e966374fc8b65d634d97d41c11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:12.893493 25638 consensus_queue.cc:1035] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.894992 25605 consensus_queue.cc:1035] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.896288 25634 consensus_queue.cc:1035] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.902191 25641 consensus_queue.cc:1035] T b7a899d61a8b481eb8f02bd4f1997a05 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.905151 25641 consensus_queue.cc:1035] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.910988 25636 consensus_queue.cc:1035] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:11:12.917627 25641 consensus_queue.cc:1035] T 6175bd8e1b7344d086a6ad2d59b50841 P 9250c2e966374fc8b65d634d97d41c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf64abf9ec642e0acb49eb700e54205" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.944270 25647 consensus_queue.cc:1035] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.944764 25647 consensus_queue.cc:1035] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.955181 25636 consensus_queue.cc:1035] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:12.965044 25634 consensus_queue.cc:1035] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:13.000151 25669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:13.000514 25669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:13.007843 25669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:11:13.057891 25634 consensus_queue.cc:1035] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d5514a6364453581ede69f4179186b" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:13.063108 25636 consensus_queue.cc:1035] T 9fe51bafa1a24fe0bd34f0d2333ab9b5 P baf64abf9ec642e0acb49eb700e54205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250c2e966374fc8b65d634d97d41c11" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 343.616 ms
time per row: 0.171808 ms
I20240503 08:11:13.985203 25478 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:13.990074 25353 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:13.991670 25218 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:14.158433 25478 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:14.159394 25218 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:14.161794 25353 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:14.347123 25478 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:14.356568 25218 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:14.359163 25353 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:14.980959 25478 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:14.988795 25353 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:14.991783 25218 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:15.155431 25478 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:11:15.155712 25218 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:15.156857 25353 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:15.303025 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 25170
I20240503 08:11:15.309650 25292 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:15.434518 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 25170
W20240503 08:11:15.451776 25315 connection.cc:535] server connection from 127.24.123.1:39703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:15.451803 25097 connection.cc:535] server connection from 127.24.123.1:53447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:15.452069 25316 connection.cc:535] client connection to 127.24.123.1:36251 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:15.452049 25446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.123.1:36251 (error 108)
W20240503 08:11:15.452288 25316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:15.452878 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 25300
W20240503 08:11:15.454358 25446 consensus_peers.cc:485] T ba38aecd198c4920942d189fbfe49653 P 89d5514a6364453581ede69f4179186b -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:15.454506 25446 consensus_peers.cc:485] T f7ea4abd02e24b98935c3e614c2b7de8 P 89d5514a6364453581ede69f4179186b -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:15.454595 25446 consensus_peers.cc:485] T 53f0fba475d84890a0b1496fdbe0ba28 P 89d5514a6364453581ede69f4179186b -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:15.454690 25446 consensus_peers.cc:485] T 48a82bdbebae44489ec337f79c21507e P 89d5514a6364453581ede69f4179186b -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:15.454941 25316 consensus_peers.cc:485] T 2291b936dd324f0fbaa0ff23b1369447 P baf64abf9ec642e0acb49eb700e54205 -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:15.455055 25316 consensus_peers.cc:485] T 286d705044a24c9b995fa8795df2205a P baf64abf9ec642e0acb49eb700e54205 -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:15.455154 25316 consensus_peers.cc:485] T cc7fdef04645494e8569400bb5ae66ce P baf64abf9ec642e0acb49eb700e54205 -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:15.455231 25316 consensus_peers.cc:485] T 92fda691c4a54496b216ca52582b3aa2 P baf64abf9ec642e0acb49eb700e54205 -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:15.455333 25316 consensus_peers.cc:485] T f4d5437b33ba439f86ded3348da6159a P baf64abf9ec642e0acb49eb700e54205 -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:15.459738 25422 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:11:15.581194 25316 consensus_peers.cc:485] T 06530128c24242828d4a602f75b6bfe7 P baf64abf9ec642e0acb49eb700e54205 -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:15.581605 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 25300
W20240503 08:11:15.584059 25446 consensus_peers.cc:485] T 44dfb618fbfc4e05bd57d7b6a97bb7c3 P 89d5514a6364453581ede69f4179186b -> Peer 9250c2e966374fc8b65d634d97d41c11 (127.24.123.1:36251): Couldn't send request to peer 9250c2e966374fc8b65d634d97d41c11. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:36251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:15.598377 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 25430
I20240503 08:11:15.604357 25552 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:15.725273 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 25430
I20240503 08:11:15.741950 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 25080
I20240503 08:11:15.743052 25141 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:15.856307 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 25080
2024-05-03T08:11:15Z chronyd exiting
[ OK ] TableRangeRebalancingTest.Basic (9655 ms)
[----------] 1 test from TableRangeRebalancingTest (9655 ms total)
[----------] 1 test from RebalanceStartSafetyTest
[ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1
I20240503 08:11:15.903071 25068 test_util.cc:268] Using random seed: 1132648003
I20240503 08:11:15.905571 25068 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:15.905637 25068 ts_itest-base.cc:113] --------------
I20240503 08:11:15.905683 25068 ts_itest-base.cc:114] 5 tablet servers
I20240503 08:11:15.905728 25068 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:15.905786 25068 ts_itest-base.cc:116] --------------
2024-05-03T08:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:15Z Disabled control of system clock
I20240503 08:11:15.925987 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:35519
--webserver_interface=127.24.123.62
--webserver_port=0
--builtin_ntp_servers=127.24.123.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.123.62:35519
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:11:16.226889 25936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:16.227191 25936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:16.227324 25936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:16.234520 25936 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:16.234671 25936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:16.234742 25936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:16.234822 25936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:16.234899 25936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:16.243747 25936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.123.62:35519
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.123.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:16.245018 25936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:16.246662 25936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:16.257355 25941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:16.257992 25944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:16.258606 25942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:16.259267 25936 server_base.cc:1034] running on GCE node
I20240503 08:11:16.259935 25936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:16.261227 25936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:16.262410 25936 hybrid_clock.cc:648] HybridClock initialized: now 1714723876262368 us; error 34 us; skew 500 ppm
I20240503 08:11:16.262724 25936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:16.264832 25936 webserver.cc:460] Webserver started at http://127.24.123.62:40529/ using document root <none> and password file <none>
I20240503 08:11:16.265481 25936 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:16.265587 25936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:16.265892 25936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:16.268395 25936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f4d6e693f3954ef89de498ddc5382ea1"
format_stamp: "Formatted at 2024-05-03 08:11:16 on dist-test-slave-gkrw"
I20240503 08:11:16.269127 25936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f4d6e693f3954ef89de498ddc5382ea1"
format_stamp: "Formatted at 2024-05-03 08:11:16 on dist-test-slave-gkrw"
I20240503 08:11:16.273977 25936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 08:11:16.277145 25950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:16.278438 25936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:16.278605 25936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f4d6e693f3954ef89de498ddc5382ea1"
format_stamp: "Formatted at 2024-05-03 08:11:16 on dist-test-slave-gkrw"
I20240503 08:11:16.278784 25936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:16.294288 25936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:16.295229 25936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:16.295521 25936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:16.311831 25936 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.62:35519
I20240503 08:11:16.311851 26001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.62:35519 every 8 connection(s)
I20240503 08:11:16.313103 25936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:16.316659 26002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:16.319340 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 25936
I20240503 08:11:16.319514 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:16.327564 26002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1: Bootstrap starting.
I20240503 08:11:16.330866 26002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:16.331902 26002 log.cc:824] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:16.334228 26002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1: No bootstrap required, opened a new log
I20240503 08:11:16.340214 26002 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d6e693f3954ef89de498ddc5382ea1" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 35519 } }
I20240503 08:11:16.340529 26002 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:16.340623 26002 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d6e693f3954ef89de498ddc5382ea1, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.341136 26002 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d6e693f3954ef89de498ddc5382ea1" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 35519 } }
I20240503 08:11:16.341331 26002 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:16.341450 26002 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:16.341583 26002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:16.404441 26002 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d6e693f3954ef89de498ddc5382ea1" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 35519 } }
I20240503 08:11:16.405133 26002 leader_election.cc:302] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4d6e693f3954ef89de498ddc5382ea1; no voters:
I20240503 08:11:16.405853 26002 leader_election.cc:288] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:16.405961 26007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:16.407088 26007 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [term 1 LEADER]: Becoming Leader. State: Replica: f4d6e693f3954ef89de498ddc5382ea1, State: Running, Role: LEADER
I20240503 08:11:16.407663 26007 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d6e693f3954ef89de498ddc5382ea1" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 35519 } }
I20240503 08:11:16.408017 26002 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:16.410300 26008 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4d6e693f3954ef89de498ddc5382ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d6e693f3954ef89de498ddc5382ea1" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 35519 } } }
I20240503 08:11:16.410641 26008 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:16.411007 26009 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4d6e693f3954ef89de498ddc5382ea1. Latest consensus state: current_term: 1 leader_uuid: "f4d6e693f3954ef89de498ddc5382ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d6e693f3954ef89de498ddc5382ea1" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 35519 } } }
I20240503 08:11:16.411340 26009 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:16.412350 26016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:16.416755 26016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:16.424993 26016 catalog_manager.cc:1347] Generated new cluster ID: 579bfa7a0c9d49ec8d7c883bebf2a277
I20240503 08:11:16.425100 26016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:16.436800 26016 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:16.437933 26016 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:16.446456 26016 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f4d6e693f3954ef89de498ddc5382ea1: Generated new TSK 0
I20240503 08:11:16.447388 26016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:16.461038 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.1:0
--local_ip_for_outbound_sockets=127.24.123.1
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--builtin_ntp_servers=127.24.123.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:16.780408 26026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:16.780714 26026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:16.780839 26026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:16.788450 26026 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:16.788580 26026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:16.788725 26026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.1
I20240503 08:11:16.797613 26026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.1
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:16.798924 26026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:16.800588 26026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:16.813686 26031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:16.814252 26032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:16.814697 26034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:16.815573 26026 server_base.cc:1034] running on GCE node
I20240503 08:11:16.816252 26026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:16.817282 26026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:16.818486 26026 hybrid_clock.cc:648] HybridClock initialized: now 1714723876818433 us; error 48 us; skew 500 ppm
I20240503 08:11:16.818827 26026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:16.820936 26026 webserver.cc:460] Webserver started at http://127.24.123.1:34967/ using document root <none> and password file <none>
I20240503 08:11:16.821616 26026 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:16.821722 26026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:16.822006 26026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:16.824458 26026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "91f7805bf551481393d97aab755f9919"
format_stamp: "Formatted at 2024-05-03 08:11:16 on dist-test-slave-gkrw"
I20240503 08:11:16.825178 26026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "91f7805bf551481393d97aab755f9919"
format_stamp: "Formatted at 2024-05-03 08:11:16 on dist-test-slave-gkrw"
I20240503 08:11:16.829906 26026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 08:11:16.833016 26040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:16.834208 26026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:16.834349 26026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "91f7805bf551481393d97aab755f9919"
format_stamp: "Formatted at 2024-05-03 08:11:16 on dist-test-slave-gkrw"
I20240503 08:11:16.834534 26026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:16.871966 26026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:16.872836 26026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:16.873103 26026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:16.874120 26026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:16.875609 26026 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:16.875701 26026 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:16.875811 26026 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:16.875878 26026 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:16.904275 26026 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.1:33319
I20240503 08:11:16.904335 26152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.1:33319 every 8 connection(s)
I20240503 08:11:16.905542 26026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:16.907544 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 26026
I20240503 08:11:16.907732 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:16.911116 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.2:0
--local_ip_for_outbound_sockets=127.24.123.2
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--builtin_ntp_servers=127.24.123.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:16.918527 26153 heartbeater.cc:347] Connected to a master server at 127.24.123.62:35519
I20240503 08:11:16.918819 26153 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:16.919625 26153 heartbeater.cc:511] Master 127.24.123.62:35519 requested a full tablet report, sending...
I20240503 08:11:16.921821 25967 ts_manager.cc:194] Registered new tserver with Master: 91f7805bf551481393d97aab755f9919 (127.24.123.1:33319)
I20240503 08:11:16.923897 25967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.1:45623
W20240503 08:11:17.219765 26157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:17.220050 26157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:17.220137 26157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:17.227468 26157 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:17.227586 26157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:17.227707 26157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.2
I20240503 08:11:17.236629 26157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.2
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:17.237875 26157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:17.239642 26157 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.251073 26162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.251578 26163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.252203 26165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.252768 26157 server_base.cc:1034] running on GCE node
I20240503 08:11:17.253293 26157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:17.254141 26157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:17.255378 26157 hybrid_clock.cc:648] HybridClock initialized: now 1714723877255322 us; error 44 us; skew 500 ppm
I20240503 08:11:17.255717 26157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.257978 26157 webserver.cc:460] Webserver started at http://127.24.123.2:46029/ using document root <none> and password file <none>
I20240503 08:11:17.258697 26157 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.258821 26157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.259177 26157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.261770 26157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "06bcd4d6f6514b1389ac4e34d9c41f01"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-gkrw"
I20240503 08:11:17.262542 26157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "06bcd4d6f6514b1389ac4e34d9c41f01"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-gkrw"
I20240503 08:11:17.267659 26157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 08:11:17.271162 26171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.272351 26157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 08:11:17.272497 26157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "06bcd4d6f6514b1389ac4e34d9c41f01"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-gkrw"
I20240503 08:11:17.272677 26157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.295629 26157 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.296521 26157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:17.296823 26157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.297833 26157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.299582 26157 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.299690 26157 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.299805 26157 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.299880 26157 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.331409 26157 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.2:38751
I20240503 08:11:17.331449 26283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.2:38751 every 8 connection(s)
I20240503 08:11:17.332769 26157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:17.337536 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 26157
I20240503 08:11:17.337692 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:17.340616 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.3:0
--local_ip_for_outbound_sockets=127.24.123.3
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--builtin_ntp_servers=127.24.123.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:17.344535 26284 heartbeater.cc:347] Connected to a master server at 127.24.123.62:35519
I20240503 08:11:17.344794 26284 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.345516 26284 heartbeater.cc:511] Master 127.24.123.62:35519 requested a full tablet report, sending...
I20240503 08:11:17.346758 25967 ts_manager.cc:194] Registered new tserver with Master: 06bcd4d6f6514b1389ac4e34d9c41f01 (127.24.123.2:38751)
I20240503 08:11:17.347501 25967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.2:45657
W20240503 08:11:17.648700 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:17.648981 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:17.649091 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:17.656204 26287 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:17.656337 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:17.656488 26287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.3
I20240503 08:11:17.665159 26287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.3
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:17.666393 26287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:17.667989 26287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.679419 26292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.680047 26295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.680812 26293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.681404 26287 server_base.cc:1034] running on GCE node
I20240503 08:11:17.681989 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:17.682977 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:17.684165 26287 hybrid_clock.cc:648] HybridClock initialized: now 1714723877684118 us; error 39 us; skew 500 ppm
I20240503 08:11:17.684480 26287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:17.686599 26287 webserver.cc:460] Webserver started at http://127.24.123.3:35477/ using document root <none> and password file <none>
I20240503 08:11:17.687320 26287 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.687448 26287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.687739 26287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.690133 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "47e9e3939dc84057b9303f1d05c6b3fb"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-gkrw"
I20240503 08:11:17.690853 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "47e9e3939dc84057b9303f1d05c6b3fb"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-gkrw"
I20240503 08:11:17.695680 26287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:11:17.698655 26301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.699873 26287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 08:11:17.700011 26287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "47e9e3939dc84057b9303f1d05c6b3fb"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-gkrw"
I20240503 08:11:17.700181 26287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.731671 26287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.732507 26287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:17.732766 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.733695 26287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.735159 26287 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.735252 26287 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.735361 26287 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.735425 26287 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.763689 26287 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.3:43541
I20240503 08:11:17.763753 26413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.3:43541 every 8 connection(s)
I20240503 08:11:17.764950 26287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:17.766561 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 26287
I20240503 08:11:17.766728 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:17.770138 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.4:0
--local_ip_for_outbound_sockets=127.24.123.4
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--builtin_ntp_servers=127.24.123.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:17.778074 26414 heartbeater.cc:347] Connected to a master server at 127.24.123.62:35519
I20240503 08:11:17.778359 26414 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.779253 26414 heartbeater.cc:511] Master 127.24.123.62:35519 requested a full tablet report, sending...
I20240503 08:11:17.780740 25967 ts_manager.cc:194] Registered new tserver with Master: 47e9e3939dc84057b9303f1d05c6b3fb (127.24.123.3:43541)
I20240503 08:11:17.781800 25967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.3:52651
I20240503 08:11:17.926673 26153 heartbeater.cc:503] Master 127.24.123.62:35519 was elected leader, sending a full tablet report...
W20240503 08:11:18.079234 26417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:18.079488 26417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:18.079620 26417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:18.086943 26417 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:18.087054 26417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:18.087210 26417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.4
I20240503 08:11:18.095791 26417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.4
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:18.096976 26417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:18.098572 26417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:18.109846 26422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.110483 26423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.111071 26425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:18.111675 26417 server_base.cc:1034] running on GCE node
I20240503 08:11:18.112210 26417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:18.113055 26417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:18.114234 26417 hybrid_clock.cc:648] HybridClock initialized: now 1714723878114192 us; error 33 us; skew 500 ppm
I20240503 08:11:18.114527 26417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:18.120142 26417 webserver.cc:460] Webserver started at http://127.24.123.4:39949/ using document root <none> and password file <none>
I20240503 08:11:18.120852 26417 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:18.120956 26417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:18.121256 26417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:18.123734 26417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8b7456d4aed04c8a8c291563980f2b10"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-gkrw"
I20240503 08:11:18.124437 26417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8b7456d4aed04c8a8c291563980f2b10"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-gkrw"
I20240503 08:11:18.129113 26417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:11:18.132155 26431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.133335 26417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:18.133481 26417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8b7456d4aed04c8a8c291563980f2b10"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-gkrw"
I20240503 08:11:18.133656 26417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:18.169591 26417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:18.170423 26417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:18.170686 26417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:18.171653 26417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:18.173059 26417 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:18.173141 26417 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.173246 26417 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:18.173312 26417 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.201911 26417 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.4:45337
I20240503 08:11:18.201969 26543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.4:45337 every 8 connection(s)
I20240503 08:11:18.203225 26417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:18.207194 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 26417
I20240503 08:11:18.207346 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:18.210467 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.5:0
--local_ip_for_outbound_sockets=127.24.123.5
--webserver_interface=127.24.123.5
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--builtin_ntp_servers=127.24.123.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:18.216133 26544 heartbeater.cc:347] Connected to a master server at 127.24.123.62:35519
I20240503 08:11:18.216418 26544 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:18.217187 26544 heartbeater.cc:511] Master 127.24.123.62:35519 requested a full tablet report, sending...
I20240503 08:11:18.218554 25967 ts_manager.cc:194] Registered new tserver with Master: 8b7456d4aed04c8a8c291563980f2b10 (127.24.123.4:45337)
I20240503 08:11:18.219619 25967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.4:44735
I20240503 08:11:18.349769 26284 heartbeater.cc:503] Master 127.24.123.62:35519 was elected leader, sending a full tablet report...
W20240503 08:11:18.524329 26547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:18.524612 26547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:18.524699 26547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:18.531868 26547 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:18.531982 26547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:18.532104 26547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.5
I20240503 08:11:18.540771 26547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.123.5
--webserver_port=0
--tserver_master_addrs=127.24.123.62:35519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.5
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:18.541949 26547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:18.543581 26547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:18.555016 26555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:18.555768 26547 server_base.cc:1034] running on GCE node
W20240503 08:11:18.556193 26553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.556603 26552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:18.557225 26547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:18.558074 26547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:18.559255 26547 hybrid_clock.cc:648] HybridClock initialized: now 1714723878559214 us; error 33 us; skew 500 ppm
I20240503 08:11:18.559556 26547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:18.561590 26547 webserver.cc:460] Webserver started at http://127.24.123.5:37561/ using document root <none> and password file <none>
I20240503 08:11:18.562260 26547 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:18.562361 26547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:18.562659 26547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:18.565124 26547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-gkrw"
I20240503 08:11:18.565932 26547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-gkrw"
I20240503 08:11:18.571273 26547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240503 08:11:18.574378 26561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.575713 26547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:18.575858 26547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-gkrw"
I20240503 08:11:18.576028 26547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:18.605109 26547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:18.606166 26547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:18.606493 26547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:18.607439 26547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:18.608860 26547 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:18.608942 26547 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.609020 26547 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:18.609071 26547 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.636982 26547 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.5:41189
I20240503 08:11:18.637038 26673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.5:41189 every 8 connection(s)
I20240503 08:11:18.638258 26547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:18.647003 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 26547
I20240503 08:11:18.647195 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:18.650406 26674 heartbeater.cc:347] Connected to a master server at 127.24.123.62:35519
I20240503 08:11:18.650700 26674 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:18.651507 26674 heartbeater.cc:511] Master 127.24.123.62:35519 requested a full tablet report, sending...
I20240503 08:11:18.652758 25967 ts_manager.cc:194] Registered new tserver with Master: f40c0042c6ae4cb0a03d7a6c9c2ef5ab (127.24.123.5:41189)
I20240503 08:11:18.653470 25967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.5:38567
I20240503 08:11:18.662639 25068 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:18.677150 25967 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57136:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:18.722074 26609 tablet_service.cc:1467] Processing CreateTablet for tablet 63e5d0a657ef4d54bd4b46aae513ebd8 (DEFAULT_TABLE table=TestTable [id=dbbf71c06ab14b67946b810d332d57e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.722213 26088 tablet_service.cc:1467] Processing CreateTablet for tablet 63e5d0a657ef4d54bd4b46aae513ebd8 (DEFAULT_TABLE table=TestTable [id=dbbf71c06ab14b67946b810d332d57e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.722548 26219 tablet_service.cc:1467] Processing CreateTablet for tablet 63e5d0a657ef4d54bd4b46aae513ebd8 (DEFAULT_TABLE table=TestTable [id=dbbf71c06ab14b67946b810d332d57e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:18.723464 26609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e5d0a657ef4d54bd4b46aae513ebd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.723685 26219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e5d0a657ef4d54bd4b46aae513ebd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.723722 26088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e5d0a657ef4d54bd4b46aae513ebd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.732962 26692 tablet_bootstrap.cc:492] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab: Bootstrap starting.
I20240503 08:11:18.736183 26693 tablet_bootstrap.cc:492] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01: Bootstrap starting.
I20240503 08:11:18.737021 26692 tablet_bootstrap.cc:654] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.737497 26694 tablet_bootstrap.cc:492] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919: Bootstrap starting.
I20240503 08:11:18.738428 26692 log.cc:824] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:18.739573 26693 tablet_bootstrap.cc:654] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.740384 26694 tablet_bootstrap.cc:654] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.740945 26693 log.cc:824] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:18.741328 26694 log.cc:824] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:18.741680 26692 tablet_bootstrap.cc:492] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab: No bootstrap required, opened a new log
I20240503 08:11:18.741956 26692 ts_tablet_manager.cc:1367] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20240503 08:11:18.743657 26694 tablet_bootstrap.cc:492] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919: No bootstrap required, opened a new log
I20240503 08:11:18.743878 26694 ts_tablet_manager.cc:1367] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240503 08:11:18.744639 26693 tablet_bootstrap.cc:492] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01: No bootstrap required, opened a new log
I20240503 08:11:18.744886 26693 ts_tablet_manager.cc:1367] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240503 08:11:18.749385 26694 raft_consensus.cc:348] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } }
I20240503 08:11:18.749913 26694 raft_consensus.cc:374] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.750015 26694 raft_consensus.cc:729] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f7805bf551481393d97aab755f9919, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.750250 26692 raft_consensus.cc:348] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } }
I20240503 08:11:18.750670 26694 consensus_queue.cc:260] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } }
I20240503 08:11:18.750787 26692 raft_consensus.cc:374] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.750905 26692 raft_consensus.cc:729] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40c0042c6ae4cb0a03d7a6c9c2ef5ab, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.751644 26692 consensus_queue.cc:260] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } }
I20240503 08:11:18.752307 26694 ts_tablet_manager.cc:1398] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 08:11:18.752676 26674 heartbeater.cc:503] Master 127.24.123.62:35519 was elected leader, sending a full tablet report...
I20240503 08:11:18.753182 26692 ts_tablet_manager.cc:1398] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20240503 08:11:18.754640 26693 raft_consensus.cc:348] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } }
I20240503 08:11:18.754985 26693 raft_consensus.cc:374] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.755075 26693 raft_consensus.cc:729] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06bcd4d6f6514b1389ac4e34d9c41f01, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.755710 26693 consensus_queue.cc:260] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } }
I20240503 08:11:18.757258 26693 ts_tablet_manager.cc:1398] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20240503 08:11:18.784421 26414 heartbeater.cc:503] Master 127.24.123.62:35519 was elected leader, sending a full tablet report...
I20240503 08:11:18.793071 26700 raft_consensus.cc:482] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:18.793305 26700 raft_consensus.cc:504] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } }
I20240503 08:11:18.794910 26700 leader_election.cc:288] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40c0042c6ae4cb0a03d7a6c9c2ef5ab (127.24.123.5:41189), 91f7805bf551481393d97aab755f9919 (127.24.123.1:33319)
I20240503 08:11:18.801339 26629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e5d0a657ef4d54bd4b46aae513ebd8" candidate_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" is_pre_election: true
I20240503 08:11:18.801398 26108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e5d0a657ef4d54bd4b46aae513ebd8" candidate_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f7805bf551481393d97aab755f9919" is_pre_election: true
I20240503 08:11:18.801791 26629 raft_consensus.cc:2454] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06bcd4d6f6514b1389ac4e34d9c41f01 in term 0.
I20240503 08:11:18.801800 26108 raft_consensus.cc:2454] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06bcd4d6f6514b1389ac4e34d9c41f01 in term 0.
I20240503 08:11:18.802335 26174 leader_election.cc:302] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06bcd4d6f6514b1389ac4e34d9c41f01, f40c0042c6ae4cb0a03d7a6c9c2ef5ab; no voters:
I20240503 08:11:18.802687 26700 raft_consensus.cc:2781] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:18.802836 26700 raft_consensus.cc:482] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:18.802961 26700 raft_consensus.cc:3037] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.805006 26700 raft_consensus.cc:504] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } }
I20240503 08:11:18.805524 26700 leader_election.cc:288] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [CANDIDATE]: Term 1 election: Requested vote from peers f40c0042c6ae4cb0a03d7a6c9c2ef5ab (127.24.123.5:41189), 91f7805bf551481393d97aab755f9919 (127.24.123.1:33319)
I20240503 08:11:18.805918 26629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e5d0a657ef4d54bd4b46aae513ebd8" candidate_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab"
I20240503 08:11:18.805958 26108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e5d0a657ef4d54bd4b46aae513ebd8" candidate_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f7805bf551481393d97aab755f9919"
I20240503 08:11:18.806082 26629 raft_consensus.cc:3037] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.806113 26108 raft_consensus.cc:3037] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.808336 26629 raft_consensus.cc:2454] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06bcd4d6f6514b1389ac4e34d9c41f01 in term 1.
I20240503 08:11:18.808336 26108 raft_consensus.cc:2454] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06bcd4d6f6514b1389ac4e34d9c41f01 in term 1.
I20240503 08:11:18.808745 26175 leader_election.cc:302] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06bcd4d6f6514b1389ac4e34d9c41f01, 91f7805bf551481393d97aab755f9919; no voters:
I20240503 08:11:18.809118 26700 raft_consensus.cc:2781] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:18.809588 26700 raft_consensus.cc:686] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [term 1 LEADER]: Becoming Leader. State: Replica: 06bcd4d6f6514b1389ac4e34d9c41f01, State: Running, Role: LEADER
I20240503 08:11:18.810127 26700 consensus_queue.cc:237] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } }
I20240503 08:11:18.814965 25967 catalog_manager.cc:5484] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06bcd4d6f6514b1389ac4e34d9c41f01 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06bcd4d6f6514b1389ac4e34d9c41f01" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 38751 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:18.835139 26285 tablet.cc:2369] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:18.839452 25068 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:18.841873 25068 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 91f7805bf551481393d97aab755f9919 to finish bootstrapping
I20240503 08:11:18.850075 25068 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 06bcd4d6f6514b1389ac4e34d9c41f01 to finish bootstrapping
I20240503 08:11:18.856384 25068 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 47e9e3939dc84057b9303f1d05c6b3fb to finish bootstrapping
I20240503 08:11:18.864542 25068 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8b7456d4aed04c8a8c291563980f2b10 to finish bootstrapping
I20240503 08:11:18.872251 25068 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f40c0042c6ae4cb0a03d7a6c9c2ef5ab to finish bootstrapping
W20240503 08:11:18.889814 26675 tablet.cc:2369] T 63e5d0a657ef4d54bd4b46aae513ebd8 P f40c0042c6ae4cb0a03d7a6c9c2ef5ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:18.908298 26154 tablet.cc:2369] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 91f7805bf551481393d97aab755f9919: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:19.221936 26544 heartbeater.cc:503] Master 127.24.123.62:35519 was elected leader, sending a full tablet report...
I20240503 08:11:19.298135 26219 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:19.305477 26088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.307003 26479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.309691 26349 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.337318 26609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.391865 26743 consensus_queue.cc:1035] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40c0042c6ae4cb0a03d7a6c9c2ef5ab" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:19.400610 26743 consensus_queue.cc:1035] T 63e5d0a657ef4d54bd4b46aae513ebd8 P 06bcd4d6f6514b1389ac4e34d9c41f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f7805bf551481393d97aab755f9919" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 33319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:19.494387 26219 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:19.506945 26479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.510425 26609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.516103 26349 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.517383 26088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.694711 26219 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:19.706475 26349 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.713270 26088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.716509 26609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:19.728453 26479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.299504 26219 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:20.314414 26479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.315287 26349 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.327613 26609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.341464 26088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.486253 26219 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:20.486495 26349 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.502812 26479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.509671 26609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.521817 26088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.668962 26219 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:20.687852 26479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.690297 26349 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.701745 26609 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.703299 26088 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:20.860930 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 26026
I20240503 08:11:20.862205 26147 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:20.956079 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 26026
I20240503 08:11:20.969838 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 26157
I20240503 08:11:20.970819 26279 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:21.065122 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 26157
I20240503 08:11:21.079361 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 26287
I20240503 08:11:21.080339 26408 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:21.169132 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 26287
I20240503 08:11:21.182336 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 26417
I20240503 08:11:21.183353 26539 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:21.272653 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 26417
W20240503 08:11:21.286049 25951 connection.cc:535] server connection from 127.24.123.4:44735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:21.286276 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 26547
I20240503 08:11:21.287325 26668 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:21.379268 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 26547
I20240503 08:11:21.392766 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 25936
I20240503 08:11:21.393801 25997 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:21.506544 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 25936
2024-05-03T08:11:21Z chronyd exiting
[ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (5640 ms)
[----------] 1 test from RebalanceStartSafetyTest (5640 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/7
I20240503 08:11:21.543579 25068 test_util.cc:268] Using random seed: 1138288501
I20240503 08:11:21.545269 25068 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:21.545343 25068 ts_itest-base.cc:113] --------------
I20240503 08:11:21.545384 25068 ts_itest-base.cc:114] 7 tablet servers
I20240503 08:11:21.545432 25068 ts_itest-base.cc:115] 5 replicas per TS
I20240503 08:11:21.545469 25068 ts_itest-base.cc:116] --------------
2024-05-03T08:11:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:21Z Disabled control of system clock
I20240503 08:11:21.565301 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:40087
--webserver_interface=127.24.123.62
--webserver_port=0
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.123.62:40087
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 08:11:21.873199 26932 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:11:21.873499 26932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:21.873600 26932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:21.873692 26932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:21.880975 26932 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:21.881076 26932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:21.881160 26932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:21.881214 26932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:21.881291 26932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:21.890208 26932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.123.62:40087
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:40087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.123.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:21.891512 26932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:21.893194 26932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:21.904201 26938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:21.904845 26940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:21.905200 26937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:21.905920 26932 server_base.cc:1034] running on GCE node
I20240503 08:11:21.906666 26932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:21.908202 26932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:21.909435 26932 hybrid_clock.cc:648] HybridClock initialized: now 1714723881909369 us; error 35 us; skew 500 ppm
I20240503 08:11:21.909791 26932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:21.911931 26932 webserver.cc:460] Webserver started at http://127.24.123.62:43343/ using document root <none> and password file <none>
I20240503 08:11:21.912585 26932 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:21.912708 26932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:21.912981 26932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:21.915508 26932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "140b761e00e94e39b950d32863080556"
format_stamp: "Formatted at 2024-05-03 08:11:21 on dist-test-slave-gkrw"
I20240503 08:11:21.916195 26932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "140b761e00e94e39b950d32863080556"
format_stamp: "Formatted at 2024-05-03 08:11:21 on dist-test-slave-gkrw"
I20240503 08:11:21.920957 26932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:11:21.924182 26946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:21.925484 26932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:21.925645 26932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
uuid: "140b761e00e94e39b950d32863080556"
format_stamp: "Formatted at 2024-05-03 08:11:21 on dist-test-slave-gkrw"
I20240503 08:11:21.925879 26932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:21.950310 26932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:21.951167 26932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:21.951474 26932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:21.969570 26932 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.62:40087
I20240503 08:11:21.969599 26997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.62:40087 every 8 connection(s)
I20240503 08:11:21.971000 26932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:21.974665 26998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:21.979200 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 26932
I20240503 08:11:21.979413 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:21.984016 26998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556: Bootstrap starting.
I20240503 08:11:21.987200 26998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:21.988122 26998 log.cc:824] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:21.990485 26998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556: No bootstrap required, opened a new log
I20240503 08:11:21.996781 26998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140b761e00e94e39b950d32863080556" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40087 } }
I20240503 08:11:21.997085 26998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:21.997179 26998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140b761e00e94e39b950d32863080556, State: Initialized, Role: FOLLOWER
I20240503 08:11:21.997745 26998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140b761e00e94e39b950d32863080556" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40087 } }
I20240503 08:11:21.997923 26998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:21.998032 26998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:21.998167 26998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:22.025131 26998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140b761e00e94e39b950d32863080556" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40087 } }
I20240503 08:11:22.025828 26998 leader_election.cc:302] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 140b761e00e94e39b950d32863080556; no voters:
I20240503 08:11:22.026512 26998 leader_election.cc:288] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:22.026646 27003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:22.027652 27003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [term 1 LEADER]: Becoming Leader. State: Replica: 140b761e00e94e39b950d32863080556, State: Running, Role: LEADER
I20240503 08:11:22.028272 27003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140b761e00e94e39b950d32863080556" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40087 } }
I20240503 08:11:22.028529 26998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:22.031117 27005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 140b761e00e94e39b950d32863080556. Latest consensus state: current_term: 1 leader_uuid: "140b761e00e94e39b950d32863080556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140b761e00e94e39b950d32863080556" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40087 } } }
I20240503 08:11:22.031159 27004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "140b761e00e94e39b950d32863080556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140b761e00e94e39b950d32863080556" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40087 } } }
I20240503 08:11:22.031458 27004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:22.031462 27005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:22.033171 27010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:22.037688 27010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:22.045413 27010 catalog_manager.cc:1347] Generated new cluster ID: 8fa1067803964dfe823bad778080e516
I20240503 08:11:22.045522 27010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:22.065742 27010 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:22.066846 27010 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:22.076575 27010 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 140b761e00e94e39b950d32863080556: Generated new TSK 0
I20240503 08:11:22.077579 27010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:22.083324 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.1:0
--local_ip_for_outbound_sockets=127.24.123.1
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:22.395236 27022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:22.395628 27022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:22.395819 27022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:22.403635 27022 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:22.403764 27022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:22.403893 27022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.1
I20240503 08:11:22.413841 27022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.1
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:22.415158 27022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:22.416858 27022 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:22.428915 27027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:22.429354 27028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:22.430016 27030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:22.430631 27022 server_base.cc:1034] running on GCE node
I20240503 08:11:22.431281 27022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:22.432220 27022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:22.433425 27022 hybrid_clock.cc:648] HybridClock initialized: now 1714723882433374 us; error 32 us; skew 500 ppm
I20240503 08:11:22.433725 27022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:22.435865 27022 webserver.cc:460] Webserver started at http://127.24.123.1:40779/ using document root <none> and password file <none>
I20240503 08:11:22.436604 27022 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:22.436712 27022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:22.437007 27022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:22.440943 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
format_stamp: "Formatted at 2024-05-03 08:11:22 on dist-test-slave-gkrw"
I20240503 08:11:22.441673 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
format_stamp: "Formatted at 2024-05-03 08:11:22 on dist-test-slave-gkrw"
I20240503 08:11:22.446437 27022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20240503 08:11:22.449530 27036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:22.450703 27022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:22.450837 27022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
format_stamp: "Formatted at 2024-05-03 08:11:22 on dist-test-slave-gkrw"
I20240503 08:11:22.451056 27022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:22.471307 27022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:22.472103 27022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:22.472362 27022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:22.473244 27022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:22.474654 27022 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:22.474747 27022 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:22.474853 27022 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:22.474939 27022 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 08:11:22.505029 27022 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.1:39261
I20240503 08:11:22.505070 27148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.1:39261 every 8 connection(s)
I20240503 08:11:22.506247 27022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:22.511198 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 27022
I20240503 08:11:22.511366 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:22.514561 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.2:0
--local_ip_for_outbound_sockets=127.24.123.2
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:22.518604 27149 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40087
I20240503 08:11:22.518952 27149 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:22.519881 27149 heartbeater.cc:511] Master 127.24.123.62:40087 requested a full tablet report, sending...
I20240503 08:11:22.521981 26963 ts_manager.cc:194] Registered new tserver with Master: f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:22.523969 26963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.1:46077
W20240503 08:11:22.821542 27153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:22.821815 27153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:22.821899 27153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:22.829107 27153 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:22.829216 27153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:22.829376 27153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.2
I20240503 08:11:22.838176 27153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.2
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:22.839426 27153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:22.841040 27153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:22.854048 27159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:22.854651 27158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:22.855157 27161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:22.855866 27153 server_base.cc:1034] running on GCE node
I20240503 08:11:22.856499 27153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:22.857410 27153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:22.858592 27153 hybrid_clock.cc:648] HybridClock initialized: now 1714723882858531 us; error 55 us; skew 500 ppm
I20240503 08:11:22.858892 27153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:22.860972 27153 webserver.cc:460] Webserver started at http://127.24.123.2:35963/ using document root <none> and password file <none>
I20240503 08:11:22.861662 27153 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:22.861760 27153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:22.862051 27153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:22.864547 27153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "36fc7a106457453ba91f05b78a5d342b"
format_stamp: "Formatted at 2024-05-03 08:11:22 on dist-test-slave-gkrw"
I20240503 08:11:22.865249 27153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "36fc7a106457453ba91f05b78a5d342b"
format_stamp: "Formatted at 2024-05-03 08:11:22 on dist-test-slave-gkrw"
I20240503 08:11:22.869884 27153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240503 08:11:22.872970 27167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:22.874167 27153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:22.874307 27153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "36fc7a106457453ba91f05b78a5d342b"
format_stamp: "Formatted at 2024-05-03 08:11:22 on dist-test-slave-gkrw"
I20240503 08:11:22.874495 27153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:22.890941 27153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:22.891768 27153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:22.892025 27153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:22.892906 27153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:22.894282 27153 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:22.894372 27153 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:22.894490 27153 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:22.894557 27153 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:22.922441 27153 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.2:33503
I20240503 08:11:22.922489 27279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.2:33503 every 8 connection(s)
I20240503 08:11:22.923707 27153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:22.931346 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 27153
I20240503 08:11:22.931540 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:22.935091 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.3:0
--local_ip_for_outbound_sockets=127.24.123.3
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:22.936347 27280 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40087
I20240503 08:11:22.936646 27280 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:22.937415 27280 heartbeater.cc:511] Master 127.24.123.62:40087 requested a full tablet report, sending...
I20240503 08:11:22.938774 26963 ts_manager.cc:194] Registered new tserver with Master: 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503)
I20240503 08:11:22.939519 26963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.2:39081
W20240503 08:11:23.237427 27283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:23.237717 27283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:23.237828 27283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:23.245131 27283 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:23.245255 27283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:23.245368 27283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.3
I20240503 08:11:23.254290 27283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.3
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:23.255555 27283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:23.257200 27283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:23.268790 27291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:23.269291 27288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:23.269814 27289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:23.270323 27283 server_base.cc:1034] running on GCE node
I20240503 08:11:23.270852 27283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:23.271752 27283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:23.272946 27283 hybrid_clock.cc:648] HybridClock initialized: now 1714723883272900 us; error 34 us; skew 500 ppm
I20240503 08:11:23.273228 27283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:23.279397 27283 webserver.cc:460] Webserver started at http://127.24.123.3:38215/ using document root <none> and password file <none>
I20240503 08:11:23.280077 27283 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:23.280185 27283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:23.280478 27283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:23.282853 27283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fe8fdcdebbff4703a924f83646531115"
format_stamp: "Formatted at 2024-05-03 08:11:23 on dist-test-slave-gkrw"
I20240503 08:11:23.283601 27283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fe8fdcdebbff4703a924f83646531115"
format_stamp: "Formatted at 2024-05-03 08:11:23 on dist-test-slave-gkrw"
I20240503 08:11:23.288319 27283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:11:23.291357 27297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:23.292636 27283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:11:23.292783 27283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fe8fdcdebbff4703a924f83646531115"
format_stamp: "Formatted at 2024-05-03 08:11:23 on dist-test-slave-gkrw"
I20240503 08:11:23.292971 27283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:23.316226 27283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:23.317049 27283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:23.317312 27283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:23.318222 27283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:23.319671 27283 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:23.319761 27283 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:23.319867 27283 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:23.319933 27283 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:23.352218 27283 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.3:43977
I20240503 08:11:23.352262 27409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.3:43977 every 8 connection(s)
I20240503 08:11:23.353428 27283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:23.360183 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 27283
I20240503 08:11:23.360359 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:23.363291 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.4:0
--local_ip_for_outbound_sockets=127.24.123.4
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:23.365442 27410 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40087
I20240503 08:11:23.365689 27410 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:23.366417 27410 heartbeater.cc:511] Master 127.24.123.62:40087 requested a full tablet report, sending...
I20240503 08:11:23.367645 26963 ts_manager.cc:194] Registered new tserver with Master: fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:11:23.368356 26963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.3:53831
I20240503 08:11:23.526769 27149 heartbeater.cc:503] Master 127.24.123.62:40087 was elected leader, sending a full tablet report...
W20240503 08:11:23.670207 27413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:23.670491 27413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:23.670622 27413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:23.677954 27413 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:23.678079 27413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:23.678225 27413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.4
I20240503 08:11:23.687047 27413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.4
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:23.688252 27413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:23.689873 27413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:23.701020 27421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:23.701488 27419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:23.701797 27418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:23.702675 27413 server_base.cc:1034] running on GCE node
I20240503 08:11:23.703275 27413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:23.704146 27413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:23.705336 27413 hybrid_clock.cc:648] HybridClock initialized: now 1714723883705297 us; error 36 us; skew 500 ppm
I20240503 08:11:23.705631 27413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:23.707763 27413 webserver.cc:460] Webserver started at http://127.24.123.4:43375/ using document root <none> and password file <none>
I20240503 08:11:23.708449 27413 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:23.708555 27413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:23.708845 27413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:23.711337 27413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "19fb518a6ede476f809c743054941e97"
format_stamp: "Formatted at 2024-05-03 08:11:23 on dist-test-slave-gkrw"
I20240503 08:11:23.712036 27413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "19fb518a6ede476f809c743054941e97"
format_stamp: "Formatted at 2024-05-03 08:11:23 on dist-test-slave-gkrw"
I20240503 08:11:23.716711 27413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 08:11:23.719842 27427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:23.721086 27413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:11:23.721230 27413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "19fb518a6ede476f809c743054941e97"
format_stamp: "Formatted at 2024-05-03 08:11:23 on dist-test-slave-gkrw"
I20240503 08:11:23.721415 27413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:23.738818 27413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:23.739678 27413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:23.739938 27413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:23.740854 27413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:23.742254 27413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:23.742347 27413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:23.742460 27413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:23.742525 27413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:23.771683 27413 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.4:41395
I20240503 08:11:23.771720 27539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.4:41395 every 8 connection(s)
I20240503 08:11:23.773001 27413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:23.778667 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 27413
I20240503 08:11:23.778900 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:23.782290 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.5:0
--local_ip_for_outbound_sockets=127.24.123.5
--webserver_interface=127.24.123.5
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:23.786105 27540 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40087
I20240503 08:11:23.786391 27540 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:23.787216 27540 heartbeater.cc:511] Master 127.24.123.62:40087 requested a full tablet report, sending...
I20240503 08:11:23.788622 26963 ts_manager.cc:194] Registered new tserver with Master: 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:23.789675 26963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.4:48097
I20240503 08:11:23.941767 27280 heartbeater.cc:503] Master 127.24.123.62:40087 was elected leader, sending a full tablet report...
W20240503 08:11:24.097394 27543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:24.097674 27543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:24.097771 27543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:24.105304 27543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:24.105433 27543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:24.105561 27543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.5
I20240503 08:11:24.114624 27543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.123.5
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.5
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:24.115947 27543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:24.117611 27543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:24.129392 27548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.129930 27549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.130523 27551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:24.131179 27543 server_base.cc:1034] running on GCE node
I20240503 08:11:24.131704 27543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:24.132512 27543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:24.133698 27543 hybrid_clock.cc:648] HybridClock initialized: now 1714723884133657 us; error 40 us; skew 500 ppm
I20240503 08:11:24.133980 27543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:24.136073 27543 webserver.cc:460] Webserver started at http://127.24.123.5:41255/ using document root <none> and password file <none>
I20240503 08:11:24.136749 27543 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:24.136852 27543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:24.137132 27543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:24.139597 27543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "2de33eb4315b4547889d1e6edae2f2dd"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:24.140302 27543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "2de33eb4315b4547889d1e6edae2f2dd"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:24.145148 27543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:24.148413 27557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:24.149721 27543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:24.149860 27543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2de33eb4315b4547889d1e6edae2f2dd"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:24.150031 27543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:24.176651 27543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:24.177474 27543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:24.177733 27543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:24.178719 27543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:24.180210 27543 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:24.180305 27543 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:24.180415 27543 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:24.180481 27543 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:24.209671 27543 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.5:43185
I20240503 08:11:24.209738 27669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.5:43185 every 8 connection(s)
I20240503 08:11:24.210966 27543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:24.219986 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 27543
I20240503 08:11:24.220170 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:24.223058 27670 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40087
I20240503 08:11:24.223407 27670 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:24.223452 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.6:0
--local_ip_for_outbound_sockets=127.24.123.6
--webserver_interface=127.24.123.6
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:24.224401 27670 heartbeater.cc:511] Master 127.24.123.62:40087 requested a full tablet report, sending...
I20240503 08:11:24.225839 26963 ts_manager.cc:194] Registered new tserver with Master: 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:24.226589 26963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.5:35571
I20240503 08:11:24.370537 27410 heartbeater.cc:503] Master 127.24.123.62:40087 was elected leader, sending a full tablet report...
W20240503 08:11:24.529374 27673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:24.529683 27673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:24.529824 27673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:24.537192 27673 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:24.537297 27673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:24.537456 27673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.6
I20240503 08:11:24.545979 27673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.123.6
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.6
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:24.547226 27673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:24.548861 27673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:24.560241 27678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.560853 27681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.561376 27679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:24.561905 27673 server_base.cc:1034] running on GCE node
I20240503 08:11:24.562398 27673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:24.563261 27673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:24.564447 27673 hybrid_clock.cc:648] HybridClock initialized: now 1714723884564397 us; error 43 us; skew 500 ppm
I20240503 08:11:24.564724 27673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:24.566700 27673 webserver.cc:460] Webserver started at http://127.24.123.6:33637/ using document root <none> and password file <none>
I20240503 08:11:24.567382 27673 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:24.567488 27673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:24.567786 27673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:24.570102 27673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "d8d7c32cb9a241c99853a6639451c4ed"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:24.570823 27673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "d8d7c32cb9a241c99853a6639451c4ed"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:24.575397 27673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 08:11:24.578557 27687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:24.579908 27673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:24.580042 27673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d8d7c32cb9a241c99853a6639451c4ed"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:24.580210 27673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:24.593529 27673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:24.594224 27673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:24.594473 27673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:24.595337 27673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:24.596657 27673 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:24.596746 27673 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:24.596843 27673 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:24.596906 27673 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:24.624850 27673 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.6:43465
I20240503 08:11:24.624913 27799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.6:43465 every 8 connection(s)
I20240503 08:11:24.626118 27673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:24.629173 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 27673
I20240503 08:11:24.629339 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:11:24.632684 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.7:0
--local_ip_for_outbound_sockets=127.24.123.7
--webserver_interface=127.24.123.7
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:24.638748 27800 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40087
I20240503 08:11:24.639075 27800 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:24.639921 27800 heartbeater.cc:511] Master 127.24.123.62:40087 requested a full tablet report, sending...
I20240503 08:11:24.641307 26963 ts_manager.cc:194] Registered new tserver with Master: d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465)
I20240503 08:11:24.642103 26963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.6:58255
I20240503 08:11:24.792217 27540 heartbeater.cc:503] Master 127.24.123.62:40087 was elected leader, sending a full tablet report...
W20240503 08:11:24.936158 27803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:24.936448 27803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:24.936578 27803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:24.943723 27803 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:24.943845 27803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:24.943989 27803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.7
I20240503 08:11:24.952742 27803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.123.7
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.7
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:24.954052 27803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:24.955739 27803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:24.966966 27808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.967306 27811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.967685 27809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:24.968127 27803 server_base.cc:1034] running on GCE node
I20240503 08:11:24.968708 27803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:24.969830 27803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:24.971026 27803 hybrid_clock.cc:648] HybridClock initialized: now 1714723884970968 us; error 51 us; skew 500 ppm
I20240503 08:11:24.971336 27803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:24.975895 27803 webserver.cc:460] Webserver started at http://127.24.123.7:45999/ using document root <none> and password file <none>
I20240503 08:11:24.976583 27803 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:24.976701 27803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:24.976985 27803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:24.979569 27803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "1980d01e2e3e4840a8ddfca4d4e366b2"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:24.980275 27803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "1980d01e2e3e4840a8ddfca4d4e366b2"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:24.985190 27803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:24.988351 27817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:24.989570 27803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:24.989719 27803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1980d01e2e3e4840a8ddfca4d4e366b2"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:24.989893 27803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:25.010656 27803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:25.011555 27803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:25.011811 27803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:25.012673 27803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:25.014099 27803 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:25.014191 27803 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.014294 27803 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:25.014358 27803 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.043133 27803 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.7:41167
I20240503 08:11:25.043170 27929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.7:41167 every 8 connection(s)
I20240503 08:11:25.044513 27803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:25.048445 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 27803
I20240503 08:11:25.048636 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:11:25.056648 27930 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40087
I20240503 08:11:25.056914 27930 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:25.057641 27930 heartbeater.cc:511] Master 127.24.123.62:40087 requested a full tablet report, sending...
I20240503 08:11:25.058863 26963 ts_manager.cc:194] Registered new tserver with Master: 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167)
I20240503 08:11:25.059711 26963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.7:43083
I20240503 08:11:25.064180 25068 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:11:25.078493 26963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47276:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:25.112510 27475 tablet_service.cc:1467] Processing CreateTablet for tablet b8d5c3e2f8e6428aa457ba21d536cea5 (DEFAULT_TABLE table=TestTable [id=0008359f803b498b9c188b0fffbee30e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:25.114321 27475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d5c3e2f8e6428aa457ba21d536cea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:25.123693 27084 tablet_service.cc:1467] Processing CreateTablet for tablet b8d5c3e2f8e6428aa457ba21d536cea5 (DEFAULT_TABLE table=TestTable [id=0008359f803b498b9c188b0fffbee30e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:25.124939 27345 tablet_service.cc:1467] Processing CreateTablet for tablet b8d5c3e2f8e6428aa457ba21d536cea5 (DEFAULT_TABLE table=TestTable [id=0008359f803b498b9c188b0fffbee30e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:25.126350 27345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d5c3e2f8e6428aa457ba21d536cea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:25.127283 27865 tablet_service.cc:1467] Processing CreateTablet for tablet b8d5c3e2f8e6428aa457ba21d536cea5 (DEFAULT_TABLE table=TestTable [id=0008359f803b498b9c188b0fffbee30e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:25.128578 27865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d5c3e2f8e6428aa457ba21d536cea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:25.132265 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d5c3e2f8e6428aa457ba21d536cea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:25.134347 27605 tablet_service.cc:1467] Processing CreateTablet for tablet b8d5c3e2f8e6428aa457ba21d536cea5 (DEFAULT_TABLE table=TestTable [id=0008359f803b498b9c188b0fffbee30e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:25.135888 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d5c3e2f8e6428aa457ba21d536cea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:25.159559 27952 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:25.159817 27951 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:25.162003 27955 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:25.163266 27951 tablet_bootstrap.cc:654] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:25.163291 27952 tablet_bootstrap.cc:654] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:25.164525 27952 log.cc:824] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:25.164534 27951 log.cc:824] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:25.165125 27955 tablet_bootstrap.cc:654] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:25.166285 27955 log.cc:824] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:25.168197 27954 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:25.171813 27954 tablet_bootstrap.cc:654] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:25.173087 27954 log.cc:824] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:25.173282 27953 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:25.174491 27952 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:25.174793 27952 ts_tablet_manager.cc:1367] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240503 08:11:25.176003 27953 tablet_bootstrap.cc:654] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:25.176901 27953 log.cc:824] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:25.179497 27951 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:25.180795 27951 ts_tablet_manager.cc:1367] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.003s
I20240503 08:11:25.181010 27955 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:25.181340 27955 ts_tablet_manager.cc:1367] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.007s
I20240503 08:11:25.181690 27954 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: No bootstrap required, opened a new log
I20240503 08:11:25.181967 27954 ts_tablet_manager.cc:1367] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240503 08:11:25.188998 27953 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:25.189307 27953 ts_tablet_manager.cc:1367] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.008s
I20240503 08:11:25.190240 27955 raft_consensus.cc:348] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.190320 27951 raft_consensus.cc:348] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.190758 27951 raft_consensus.cc:374] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:25.190891 27951 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:25.190953 27955 raft_consensus.cc:374] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:25.191073 27955 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:25.191823 27951 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.191843 27955 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.192065 27952 raft_consensus.cc:348] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.192524 27952 raft_consensus.cc:374] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:25.192667 27952 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:25.193450 27952 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.196964 27951 ts_tablet_manager.cc:1398] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.016s user 0.007s sys 0.006s
I20240503 08:11:25.197438 27954 raft_consensus.cc:348] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.197827 27954 raft_consensus.cc:374] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:25.197908 27954 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: FOLLOWER
I20240503 08:11:25.198477 27954 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.201500 27952 ts_tablet_manager.cc:1398] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.027s user 0.010s sys 0.006s
I20240503 08:11:25.202528 27953 raft_consensus.cc:348] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.202944 27953 raft_consensus.cc:374] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:25.203044 27953 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:25.203092 27955 ts_tablet_manager.cc:1398] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.022s user 0.016s sys 0.003s
I20240503 08:11:25.203786 27953 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.204262 27930 heartbeater.cc:503] Master 127.24.123.62:40087 was elected leader, sending a full tablet report...
I20240503 08:11:25.205019 27954 ts_tablet_manager.cc:1398] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.023s user 0.008s sys 0.002s
I20240503 08:11:25.206984 27953 ts_tablet_manager.cc:1398] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.018s user 0.006s sys 0.005s
I20240503 08:11:25.207298 27670 heartbeater.cc:503] Master 127.24.123.62:40087 was elected leader, sending a full tablet report...
W20240503 08:11:25.213018 27671 tablet.cc:2369] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:25.259467 27150 tablet.cc:2369] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:25.275518 27541 tablet.cc:2369] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:25.296108 27931 tablet.cc:2369] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:25.355890 27411 tablet.cc:2369] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:25.499827 27964 raft_consensus.cc:482] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:25.500075 27964 raft_consensus.cc:504] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.503116 27964 leader_election.cc:288] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:25.509721 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:25.510288 27365 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 0.
I20240503 08:11:25.511126 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:25.511626 27104 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 0.
I20240503 08:11:25.511922 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:25.512207 27821 leader_election.cc:302] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:25.512354 27495 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 0.
I20240503 08:11:25.512586 27964 raft_consensus.cc:2781] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:25.512706 27964 raft_consensus.cc:482] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:25.512806 27964 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:25.515851 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:25.515898 27964 raft_consensus.cc:504] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.516232 27625 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 0.
I20240503 08:11:25.516825 27964 leader_election.cc:288] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 1 election: Requested vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:25.517082 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:25.517283 27365 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:25.517294 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:25.517474 27104 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:25.517450 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:25.517628 27495 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:25.519583 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:25.519757 27625 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:25.520222 27365 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 1.
I20240503 08:11:25.520378 27104 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 1.
I20240503 08:11:25.520526 27495 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 1.
I20240503 08:11:25.520694 27821 leader_election.cc:302] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:25.521011 27964 raft_consensus.cc:2781] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:25.521425 27964 raft_consensus.cc:686] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEADER]: Becoming Leader. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Running, Role: LEADER
I20240503 08:11:25.521986 27964 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:25.522446 27625 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 1.
I20240503 08:11:25.527014 26963 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7). New cstate: current_term: 1 leader_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:25.557026 25068 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:11:25.559042 25068 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f4d3aef6d5624cf4adc3859dbadccb80 to finish bootstrapping
I20240503 08:11:25.566470 25068 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 36fc7a106457453ba91f05b78a5d342b to finish bootstrapping
I20240503 08:11:25.574100 25068 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fe8fdcdebbff4703a924f83646531115 to finish bootstrapping
I20240503 08:11:25.580924 25068 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 19fb518a6ede476f809c743054941e97 to finish bootstrapping
I20240503 08:11:25.587555 25068 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2de33eb4315b4547889d1e6edae2f2dd to finish bootstrapping
I20240503 08:11:25.594341 25068 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d8d7c32cb9a241c99853a6639451c4ed to finish bootstrapping
I20240503 08:11:25.602560 25068 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1980d01e2e3e4840a8ddfca4d4e366b2 to finish bootstrapping
I20240503 08:11:25.609124 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 27803
I20240503 08:11:25.609905 27925 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:25.644547 27800 heartbeater.cc:503] Master 127.24.123.62:40087 was elected leader, sending a full tablet report...
I20240503 08:11:25.695483 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 27803
I20240503 08:11:27.074430 27975 raft_consensus.cc:482] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:27.074615 27975 raft_consensus.cc:504] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:27.076426 27975 leader_election.cc:288] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167)
W20240503 08:11:27.090042 27040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.7:41167: connect: Connection refused (error 111)
W20240503 08:11:27.092880 27040 leader_election.cc:334] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Network error: Client connection negotiation failed: client connection to 127.24.123.7:41167: connect: Connection refused (error 111)
I20240503 08:11:27.095929 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:27.096201 27365 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:27.096859 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:27.097098 27495 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:27.097493 27038 leader_election.cc:302] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 19fb518a6ede476f809c743054941e97, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters: 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:27.097911 27975 raft_consensus.cc:2781] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:11:27.098018 27975 raft_consensus.cc:482] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:27.098112 27975 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:27.098217 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:27.098439 27625 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:27.100329 27975 raft_consensus.cc:504] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:27.101125 27975 leader_election.cc:288] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 2 election: Requested vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167)
I20240503 08:11:27.101249 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:27.101330 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:27.101449 27365 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:27.101536 27495 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:11:27.102854 27040 leader_election.cc:334] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Network error: Client connection negotiation failed: client connection to 127.24.123.7:41167: connect: Connection refused (error 111)
I20240503 08:11:27.103544 27365 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 2.
I20240503 08:11:27.103735 27495 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 2.
I20240503 08:11:27.104051 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:27.104094 27038 leader_election.cc:302] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 19fb518a6ede476f809c743054941e97, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters: 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:27.104259 27625 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:27.104343 27975 raft_consensus.cc:2781] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:27.104782 27975 raft_consensus.cc:686] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 LEADER]: Becoming Leader. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Running, Role: LEADER
I20240503 08:11:27.105278 27975 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:27.106302 27625 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 2.
I20240503 08:11:27.109712 26963 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: term changed from 1 to 2, leader changed from 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) to f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1). New cstate: current_term: 2 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:27.494146 27625 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 08:11:27.495065 27975 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:27.650334 27040 consensus_peers.cc:485] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.7:41167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:27.685638 27495 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 08:11:27.686440 27975 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:27.694178 27365 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 08:11:27.695398 28001 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:29.460117 26963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47276:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:29.470443 27345 tablet_service.cc:1467] Processing CreateTablet for tablet 8f94c738c0f34d04bd933cc09c05d6a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.471004 27345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f94c738c0f34d04bd933cc09c05d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.471462 27083 tablet_service.cc:1467] Processing CreateTablet for tablet dd41f1db8043464583458568a5900a7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.471899 27083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41f1db8043464583458568a5900a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.475441 27084 tablet_service.cc:1467] Processing CreateTablet for tablet 8f94c738c0f34d04bd933cc09c05d6a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.475850 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f94c738c0f34d04bd933cc09c05d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.478511 27082 tablet_service.cc:1467] Processing CreateTablet for tablet 03b4df9c9c0c4ec59b00338de1ae2920 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.478961 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b4df9c9c0c4ec59b00338de1ae2920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.480515 27605 tablet_service.cc:1467] Processing CreateTablet for tablet dd41f1db8043464583458568a5900a7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.480924 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41f1db8043464583458568a5900a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.481087 27474 tablet_service.cc:1467] Processing CreateTablet for tablet 03b4df9c9c0c4ec59b00338de1ae2920 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.481498 27474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b4df9c9c0c4ec59b00338de1ae2920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.481735 27734 tablet_service.cc:1467] Processing CreateTablet for tablet dd41f1db8043464583458568a5900a7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.483110 27733 tablet_service.cc:1467] Processing CreateTablet for tablet 03b4df9c9c0c4ec59b00338de1ae2920 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.483419 27475 tablet_service.cc:1467] Processing CreateTablet for tablet 8f94c738c0f34d04bd933cc09c05d6a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.483786 27475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f94c738c0f34d04bd933cc09c05d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.484033 27735 tablet_service.cc:1467] Processing CreateTablet for tablet 8f94c738c0f34d04bd933cc09c05d6a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.486897 27604 tablet_service.cc:1467] Processing CreateTablet for tablet 03b4df9c9c0c4ec59b00338de1ae2920 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.487246 27604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b4df9c9c0c4ec59b00338de1ae2920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.491506 27344 tablet_service.cc:1467] Processing CreateTablet for tablet dd41f1db8043464583458568a5900a7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.491930 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41f1db8043464583458568a5900a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.495637 27734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41f1db8043464583458568a5900a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.496205 27733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b4df9c9c0c4ec59b00338de1ae2920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.498091 27735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f94c738c0f34d04bd933cc09c05d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.499982 28020 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:29.500414 28019 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:29.502089 28020 tablet_bootstrap.cc:654] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.502259 28019 tablet_bootstrap.cc:654] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.507437 28022 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:29.508028 28021 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:29.510185 28021 tablet_bootstrap.cc:654] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.512105 28019 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:29.512305 28019 ts_tablet_manager.cc:1367] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20240503 08:11:29.513567 27214 tablet_service.cc:1467] Processing CreateTablet for tablet dd41f1db8043464583458568a5900a7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.514076 28019 raft_consensus.cc:348] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.514317 28019 raft_consensus.cc:374] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.514400 28019 raft_consensus.cc:729] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.514631 28019 consensus_queue.cc:260] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.515202 28019 ts_tablet_manager.cc:1398] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:29.515537 28019 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:29.516355 28022 tablet_bootstrap.cc:654] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.516937 28019 tablet_bootstrap.cc:654] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.514226 27215 tablet_service.cc:1467] Processing CreateTablet for tablet 8f94c738c0f34d04bd933cc09c05d6a6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.518090 27215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f94c738c0f34d04bd933cc09c05d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.514413 27213 tablet_service.cc:1467] Processing CreateTablet for tablet 03b4df9c9c0c4ec59b00338de1ae2920 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:29.523730 27213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b4df9c9c0c4ec59b00338de1ae2920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.524554 27214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41f1db8043464583458568a5900a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:29.525490 28021 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:29.526554 28020 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:29.526726 28020 ts_tablet_manager.cc:1367] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.005s
I20240503 08:11:29.527709 28020 raft_consensus.cc:348] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.527962 28020 raft_consensus.cc:374] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.528056 28020 raft_consensus.cc:729] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.528302 28021 ts_tablet_manager.cc:1367] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240503 08:11:29.529567 28021 raft_consensus.cc:348] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.529837 28021 raft_consensus.cc:374] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.529924 28021 raft_consensus.cc:729] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.530239 28020 consensus_queue.cc:260] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.531652 28022 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:29.531822 28022 ts_tablet_manager.cc:1367] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240503 08:11:29.532697 28022 raft_consensus.cc:348] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.532929 28022 raft_consensus.cc:374] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.533038 28022 raft_consensus.cc:729] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.533277 28022 consensus_queue.cc:260] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.533967 28021 consensus_queue.cc:260] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.536456 28020 ts_tablet_manager.cc:1398] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20240503 08:11:29.537678 28020 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:29.540825 28028 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:29.544857 28022 ts_tablet_manager.cc:1398] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240503 08:11:29.545495 28020 tablet_bootstrap.cc:654] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.546095 28022 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:29.549934 28022 tablet_bootstrap.cc:654] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.551256 28021 ts_tablet_manager.cc:1398] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s
I20240503 08:11:29.553707 28021 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:29.555320 28021 tablet_bootstrap.cc:654] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.556145 28019 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:29.556396 28019 ts_tablet_manager.cc:1367] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.009s
I20240503 08:11:29.556814 28028 tablet_bootstrap.cc:654] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.557296 28019 raft_consensus.cc:348] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.557776 28019 raft_consensus.cc:374] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.557897 28019 raft_consensus.cc:729] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.558228 28028 log.cc:824] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:29.558225 28019 consensus_queue.cc:260] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.558946 28019 ts_tablet_manager.cc:1398] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:29.559298 28019 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:29.560925 28019 tablet_bootstrap.cc:654] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.561599 28032 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:29.562510 28020 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:29.563439 28021 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:29.563652 28021 ts_tablet_manager.cc:1367] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240503 08:11:29.564448 28020 ts_tablet_manager.cc:1367] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240503 08:11:29.564424 28021 raft_consensus.cc:348] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.564791 28021 raft_consensus.cc:374] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.564903 28021 raft_consensus.cc:729] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.565222 28021 consensus_queue.cc:260] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.565274 28020 raft_consensus.cc:348] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.565654 28020 raft_consensus.cc:374] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.565744 28020 raft_consensus.cc:729] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.566038 28020 consensus_queue.cc:260] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.566836 28032 tablet_bootstrap.cc:654] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.567577 28029 raft_consensus.cc:482] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:29.567740 28029 raft_consensus.cc:504] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.568811 28032 log.cc:824] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:29.569067 28020 ts_tablet_manager.cc:1398] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:29.569991 28029 leader_election.cc:288] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465)
I20240503 08:11:29.576195 28021 ts_tablet_manager.cc:1398] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240503 08:11:29.579344 28022 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:29.579548 28022 ts_tablet_manager.cc:1367] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20240503 08:11:29.580391 28022 raft_consensus.cc:348] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.580686 28022 raft_consensus.cc:374] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.580760 28022 raft_consensus.cc:729] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.580761 28032 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:29.582151 28032 ts_tablet_manager.cc:1367] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.008s
I20240503 08:11:29.581033 28022 consensus_queue.cc:260] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.583957 28022 ts_tablet_manager.cc:1398] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:29.600456 28028 tablet_bootstrap.cc:492] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:29.600908 28028 ts_tablet_manager.cc:1367] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.060s user 0.016s sys 0.014s
I20240503 08:11:29.604454 28032 raft_consensus.cc:348] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.604972 28032 raft_consensus.cc:374] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.605096 28032 raft_consensus.cc:729] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.605877 28032 consensus_queue.cc:260] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.606575 28031 raft_consensus.cc:482] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:29.606777 28031 raft_consensus.cc:504] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.609109 28031 leader_election.cc:288] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:29.610841 28028 raft_consensus.cc:348] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.611901 28028 raft_consensus.cc:374] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.612048 28028 raft_consensus.cc:729] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.612987 28028 consensus_queue.cc:260] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.646647 28019 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:29.646838 28019 ts_tablet_manager.cc:1367] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.010s
I20240503 08:11:29.647620 28019 raft_consensus.cc:348] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.647891 28019 raft_consensus.cc:374] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.647981 28019 raft_consensus.cc:729] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.648231 28019 consensus_queue.cc:260] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.648873 28019 ts_tablet_manager.cc:1398] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:29.662132 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:29.662395 27104 raft_consensus.cc:2454] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:29.665223 28052 raft_consensus.cc:482] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:29.665581 28052 raft_consensus.cc:504] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.678118 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:29.678695 27755 raft_consensus.cc:2454] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:29.678740 28032 ts_tablet_manager.cc:1398] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.096s user 0.007s sys 0.005s
I20240503 08:11:29.679178 28032 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:29.681633 27561 leader_election.cc:302] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, d8d7c32cb9a241c99853a6639451c4ed, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:29.682154 28029 raft_consensus.cc:2781] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:29.682271 28029 raft_consensus.cc:482] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:29.682348 28029 raft_consensus.cc:3037] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.685215 28029 raft_consensus.cc:504] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.686097 28029 leader_election.cc:288] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 election: Requested vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465)
I20240503 08:11:29.687252 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:29.687312 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:29.687512 27104 raft_consensus.cc:3037] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.687506 27755 raft_consensus.cc:3037] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:11:29.679431 27281 tablet.cc:2369] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:29.690225 27755 raft_consensus.cc:2454] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:29.690464 27104 raft_consensus.cc:2454] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:29.691222 27561 leader_election.cc:302] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, d8d7c32cb9a241c99853a6639451c4ed, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:29.691325 28032 tablet_bootstrap.cc:654] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.691589 28029 raft_consensus.cc:2781] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:29.693325 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
W20240503 08:11:29.694284 27431 leader_election.cc:341] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:29.698381 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:29.698666 27365 raft_consensus.cc:2454] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:29.698617 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:29.698863 27364 raft_consensus.cc:3037] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.694989 28052 leader_election.cc:288] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:29.697796 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:29.701896 27364 raft_consensus.cc:2454] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:29.703135 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:29.703258 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:29.703395 27235 raft_consensus.cc:2454] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
W20240503 08:11:29.703929 27428 leader_election.cc:341] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:29.706982 28029 raft_consensus.cc:686] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Becoming Leader. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Running, Role: LEADER
I20240503 08:11:29.707818 28029 consensus_queue.cc:237] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:29.709646 28028 ts_tablet_manager.cc:1398] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.109s user 0.041s sys 0.040s
I20240503 08:11:29.712730 28028 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:29.714427 28028 tablet_bootstrap.cc:654] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.717166 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:29.717404 27625 raft_consensus.cc:2454] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:29.726174 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:29.726528 27364 raft_consensus.cc:2379] T dd41f1db8043464583458568a5900a7d P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36fc7a106457453ba91f05b78a5d342b in current term 1: Already voted for candidate 2de33eb4315b4547889d1e6edae2f2dd in this term.
I20240503 08:11:29.728554 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:29.728796 27104 raft_consensus.cc:2454] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:29.729212 27431 leader_election.cc:302] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 2de33eb4315b4547889d1e6edae2f2dd, f4d3aef6d5624cf4adc3859dbadccb80; no voters: 36fc7a106457453ba91f05b78a5d342b, d8d7c32cb9a241c99853a6639451c4ed
I20240503 08:11:29.730132 28031 raft_consensus.cc:2781] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:29.730247 28031 raft_consensus.cc:482] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:29.730324 28031 raft_consensus.cc:3037] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.733155 28031 raft_consensus.cc:504] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.735380 26962 catalog_manager.cc:5484] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5). New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:29.736202 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:29.736869 28032 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:29.737087 28032 ts_tablet_manager.cc:1367] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.058s user 0.016s sys 0.003s
I20240503 08:11:29.737449 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:29.738365 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:29.738487 28031 leader_election.cc:288] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 election: Requested vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
W20240503 08:11:29.738937 27428 leader_election.cc:341] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:29.738109 28032 raft_consensus.cc:348] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
W20240503 08:11:29.739168 27431 leader_election.cc:341] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:29.739353 28032 raft_consensus.cc:374] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.739516 28032 raft_consensus.cc:729] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.739665 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:29.739869 27104 raft_consensus.cc:3037] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.739908 28032 consensus_queue.cc:260] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.740154 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:29.740377 27625 raft_consensus.cc:3037] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.740720 28032 ts_tablet_manager.cc:1398] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:11:29.741053 28032 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:29.742547 27104 raft_consensus.cc:2454] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:29.742805 28032 tablet_bootstrap.cc:654] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.742887 27625 raft_consensus.cc:2454] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:29.743238 27429 leader_election.cc:302] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 2de33eb4315b4547889d1e6edae2f2dd, f4d3aef6d5624cf4adc3859dbadccb80; no voters: 36fc7a106457453ba91f05b78a5d342b, d8d7c32cb9a241c99853a6639451c4ed
I20240503 08:11:29.744052 28031 raft_consensus.cc:2781] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:29.746801 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:29.747200 27755 raft_consensus.cc:2379] T dd41f1db8043464583458568a5900a7d P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36fc7a106457453ba91f05b78a5d342b in current term 1: Already voted for candidate 2de33eb4315b4547889d1e6edae2f2dd in this term.
I20240503 08:11:29.747269 28028 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:29.747431 28028 ts_tablet_manager.cc:1367] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.014s
I20240503 08:11:29.747512 28031 raft_consensus.cc:686] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Becoming Leader. State: Replica: 19fb518a6ede476f809c743054941e97, State: Running, Role: LEADER
I20240503 08:11:29.747879 27171 leader_election.cc:302] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 36fc7a106457453ba91f05b78a5d342b; no voters: 2de33eb4315b4547889d1e6edae2f2dd, d8d7c32cb9a241c99853a6639451c4ed, fe8fdcdebbff4703a924f83646531115
I20240503 08:11:29.748078 28031 consensus_queue.cc:237] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.748272 28052 raft_consensus.cc:3037] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.749069 28028 raft_consensus.cc:348] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.749280 28028 raft_consensus.cc:374] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.749364 28028 raft_consensus.cc:729] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.749624 28028 consensus_queue.cc:260] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.750154 28028 ts_tablet_manager.cc:1398] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:29.751004 28028 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:29.751158 28052 raft_consensus.cc:2726] T dd41f1db8043464583458568a5900a7d P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:29.752748 28028 tablet_bootstrap.cc:654] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:29.754662 28032 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:29.754906 28032 ts_tablet_manager.cc:1367] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20240503 08:11:29.755688 28032 raft_consensus.cc:348] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.755903 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41f1db8043464583458568a5900a7d" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:29.755991 28032 raft_consensus.cc:374] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.756057 28032 raft_consensus.cc:729] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.756219 27104 raft_consensus.cc:2379] T dd41f1db8043464583458568a5900a7d P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36fc7a106457453ba91f05b78a5d342b in current term 1: Already voted for candidate 2de33eb4315b4547889d1e6edae2f2dd in this term.
I20240503 08:11:29.756307 28032 consensus_queue.cc:260] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.758131 28032 ts_tablet_manager.cc:1398] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:29.758620 28028 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:29.758760 28028 ts_tablet_manager.cc:1367] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240503 08:11:29.759495 28028 raft_consensus.cc:348] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.759737 28028 raft_consensus.cc:374] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:29.759822 28028 raft_consensus.cc:729] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:29.760066 28028 consensus_queue.cc:260] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.759997 26963 catalog_manager.cc:5484] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19fb518a6ede476f809c743054941e97 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:29.760604 28028 ts_tablet_manager.cc:1398] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:29.783735 28052 raft_consensus.cc:482] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:29.783905 28052 raft_consensus.cc:504] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:29.784837 28052 leader_election.cc:288] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:29.785075 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:29.785295 27755 raft_consensus.cc:2454] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 0.
I20240503 08:11:29.785547 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:29.785866 27625 raft_consensus.cc:2379] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36fc7a106457453ba91f05b78a5d342b in current term 1: Already voted for candidate 19fb518a6ede476f809c743054941e97 in this term.
I20240503 08:11:29.785949 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:29.786222 27104 raft_consensus.cc:2379] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36fc7a106457453ba91f05b78a5d342b in current term 1: Already voted for candidate 19fb518a6ede476f809c743054941e97 in this term.
I20240503 08:11:29.790434 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b4df9c9c0c4ec59b00338de1ae2920" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:29.790947 27169 leader_election.cc:302] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 36fc7a106457453ba91f05b78a5d342b, d8d7c32cb9a241c99853a6639451c4ed; no voters: 19fb518a6ede476f809c743054941e97, 2de33eb4315b4547889d1e6edae2f2dd, f4d3aef6d5624cf4adc3859dbadccb80
I20240503 08:11:29.791235 28052 raft_consensus.cc:3037] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.792989 28052 raft_consensus.cc:2726] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:29.819763 27975 raft_consensus.cc:482] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:29.819988 27975 raft_consensus.cc:504] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.820978 27975 leader_election.cc:288] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:29.821967 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:29.822183 27495 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:29.822254 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:29.822492 27364 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:29.822950 27037 leader_election.cc:302] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:29.823200 27975 raft_consensus.cc:2781] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:29.823285 27975 raft_consensus.cc:482] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:29.823383 27975 raft_consensus.cc:3037] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.825943 27975 raft_consensus.cc:504] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.827050 27975 leader_election.cc:288] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 election: Requested vote from peers d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:29.827373 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:29.827500 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:29.827579 27364 raft_consensus.cc:3037] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.827682 27495 raft_consensus.cc:3037] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.827852 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:29.828035 27234 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:29.828433 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:29.828433 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:29.828576 27234 raft_consensus.cc:3037] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.828644 27755 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:29.828943 27754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:29.829145 27754 raft_consensus.cc:3037] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:29.830065 27364 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:29.830235 27495 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:29.830981 27038 leader_election.cc:302] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:29.831329 27975 raft_consensus.cc:2781] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:29.831343 27234 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:29.831815 27754 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:29.831815 27975 raft_consensus.cc:686] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Becoming Leader. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Running, Role: LEADER
I20240503 08:11:29.832158 27975 consensus_queue.cc:237] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:29.834942 26963 catalog_manager.cc:5484] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:29.880777 27801 tablet.cc:2369] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:30.133242 28031 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:30.163537 28073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:30.163826 28073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:30.171568 28073 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:30.175472 28029 consensus_queue.cc:1035] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 08:11:30.234830 28029 consensus_queue.cc:1035] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:30.257241 28029 consensus_queue.cc:1035] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:30.267875 28029 consensus_queue.cc:1035] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:30.285972 28031 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:30.322409 27975 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:30.367959 27753 raft_consensus.cc:3037] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:30.372869 28106 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:30.379503 27040 consensus_peers.cc:485] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.7:41167: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:30.412454 28106 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:30.465967 27975 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:30.489208 27975 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:30.490003 27975 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 465.846 ms
time per row: 0.232923 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:31.675871 26959 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47276:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:31.684744 27604 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4a8224c7c1430a9354e58bd16c7abd (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.685045 27214 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4a8224c7c1430a9354e58bd16c7abd (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.685221 27604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4a8224c7c1430a9354e58bd16c7abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.685427 27214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4a8224c7c1430a9354e58bd16c7abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.686034 27475 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4a8224c7c1430a9354e58bd16c7abd (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.686424 27475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4a8224c7c1430a9354e58bd16c7abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.688095 27734 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4a8224c7c1430a9354e58bd16c7abd (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.688501 27734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4a8224c7c1430a9354e58bd16c7abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.691498 27082 tablet_service.cc:1467] Processing CreateTablet for tablet c42929dcee3c461c8db91346ca1ab0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.691901 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42929dcee3c461c8db91346ca1ab0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.692046 27605 tablet_service.cc:1467] Processing CreateTablet for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.692261 27474 tablet_service.cc:1467] Processing CreateTablet for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.692442 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.692591 27474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.695274 27733 tablet_service.cc:1467] Processing CreateTablet for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.695407 27473 tablet_service.cc:1467] Processing CreateTablet for tablet c42929dcee3c461c8db91346ca1ab0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.695645 27733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.695832 27473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42929dcee3c461c8db91346ca1ab0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.696681 27603 tablet_service.cc:1467] Processing CreateTablet for tablet c42929dcee3c461c8db91346ca1ab0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.697078 27603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42929dcee3c461c8db91346ca1ab0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.700536 27343 tablet_service.cc:1467] Processing CreateTablet for tablet c42929dcee3c461c8db91346ca1ab0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.700703 27345 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4a8224c7c1430a9354e58bd16c7abd (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.700958 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42929dcee3c461c8db91346ca1ab0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.701119 27345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4a8224c7c1430a9354e58bd16c7abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.703567 27215 tablet_service.cc:1467] Processing CreateTablet for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.703898 28177 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:31.704005 27215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.704351 28178 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:31.705868 28178 tablet_bootstrap.cc:654] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.700546 27344 tablet_service.cc:1467] Processing CreateTablet for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.707419 27213 tablet_service.cc:1467] Processing CreateTablet for tablet c42929dcee3c461c8db91346ca1ab0f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:31.707792 27213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42929dcee3c461c8db91346ca1ab0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.708197 28177 tablet_bootstrap.cc:654] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.713707 28181 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:31.716042 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:31.716344 28181 tablet_bootstrap.cc:654] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.718611 28182 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:31.720144 28178 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:31.720955 28182 tablet_bootstrap.cc:654] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.721002 28178 ts_tablet_manager.cc:1367] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.007s
I20240503 08:11:31.723752 28184 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:31.724103 28178 raft_consensus.cc:348] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.724579 28178 raft_consensus.cc:374] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.724727 28178 raft_consensus.cc:729] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.725034 28178 consensus_queue.cc:260] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.725836 28178 ts_tablet_manager.cc:1398] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:11:31.726472 28181 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:31.726622 28181 ts_tablet_manager.cc:1367] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240503 08:11:31.729396 28184 tablet_bootstrap.cc:654] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.728312 28181 raft_consensus.cc:348] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.730329 28181 raft_consensus.cc:374] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.730434 28181 raft_consensus.cc:729] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.731173 28178 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:31.731798 28177 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:31.731961 28177 ts_tablet_manager.cc:1367] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20240503 08:11:31.732659 28177 raft_consensus.cc:348] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.732890 28177 raft_consensus.cc:374] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.732985 28177 raft_consensus.cc:729] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.733186 28178 tablet_bootstrap.cc:654] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.733219 28177 consensus_queue.cc:260] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.733695 28181 consensus_queue.cc:260] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.734277 28181 ts_tablet_manager.cc:1398] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 08:11:31.734560 28181 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:31.734908 28186 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:31.736457 28181 tablet_bootstrap.cc:654] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.737239 28182 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:31.738193 28182 ts_tablet_manager.cc:1367] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.003s
I20240503 08:11:31.738549 28186 tablet_bootstrap.cc:654] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.739044 28182 raft_consensus.cc:348] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.739229 28182 raft_consensus.cc:374] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.739430 28181 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:31.739982 28181 ts_tablet_manager.cc:1367] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 08:11:31.740187 28182 raft_consensus.cc:729] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.740234 28178 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:31.740376 28178 ts_tablet_manager.cc:1367] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 08:11:31.740597 28182 consensus_queue.cc:260] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.740887 28181 raft_consensus.cc:348] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.741043 28178 raft_consensus.cc:348] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.741333 28178 raft_consensus.cc:374] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.741453 28178 raft_consensus.cc:729] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.741611 28182 ts_tablet_manager.cc:1398] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:31.741995 28181 raft_consensus.cc:374] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.742079 28181 raft_consensus.cc:729] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.742328 28182 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:31.742308 28181 consensus_queue.cc:260] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.743891 28181 ts_tablet_manager.cc:1398] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:31.744287 28181 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:31.744638 28182 tablet_bootstrap.cc:654] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.741966 28178 consensus_queue.cc:260] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.747649 28181 tablet_bootstrap.cc:654] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.747758 28178 ts_tablet_manager.cc:1398] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:11:31.748059 28178 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:31.748617 28177 ts_tablet_manager.cc:1398] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20240503 08:11:31.749001 28177 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:31.750556 28177 tablet_bootstrap.cc:654] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.753309 28181 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:31.753512 28181 ts_tablet_manager.cc:1367] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:11:31.754329 28178 tablet_bootstrap.cc:654] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.754257 28181 raft_consensus.cc:348] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.754562 28181 raft_consensus.cc:374] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.754662 28181 raft_consensus.cc:729] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.754943 28181 consensus_queue.cc:260] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.755625 28181 ts_tablet_manager.cc:1398] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:31.756060 28186 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:31.760012 28171 raft_consensus.cc:482] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:31.760109 28177 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:31.760272 28177 ts_tablet_manager.cc:1367] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20240503 08:11:31.760180 28171 raft_consensus.cc:504] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.760358 28186 ts_tablet_manager.cc:1367] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20240503 08:11:31.761013 28177 raft_consensus.cc:348] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.761070 28186 raft_consensus.cc:348] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.761261 28177 raft_consensus.cc:374] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.761273 28186 raft_consensus.cc:374] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.761257 28171 leader_election.cc:288] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:31.761323 28177 raft_consensus.cc:729] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.761349 28186 raft_consensus.cc:729] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.761574 28186 consensus_queue.cc:260] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.761611 28177 consensus_queue.cc:260] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.762113 28177 ts_tablet_manager.cc:1398] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:31.762398 28177 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:31.762826 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:31.763031 28184 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:31.763180 28184 ts_tablet_manager.cc:1367] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.014s
I20240503 08:11:31.763689 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:31.763919 27235 raft_consensus.cc:2454] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:31.763929 28184 raft_consensus.cc:348] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.764140 28184 raft_consensus.cc:374] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.764222 28184 raft_consensus.cc:729] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.764362 28177 tablet_bootstrap.cc:654] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.764464 28184 consensus_queue.cc:260] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.764989 28184 ts_tablet_manager.cc:1398] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240503 08:11:31.765014 27431 leader_election.cc:341] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:31.765805 28178 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:31.765990 28178 ts_tablet_manager.cc:1367] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 08:11:31.765899 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:31.766775 28178 raft_consensus.cc:348] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.767077 28178 raft_consensus.cc:374] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.767110 27625 raft_consensus.cc:2454] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:31.767170 28178 raft_consensus.cc:729] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.767421 28178 consensus_queue.cc:260] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.767947 28145 raft_consensus.cc:482] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:31.768074 28145 raft_consensus.cc:504] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.768957 28145 leader_election.cc:288] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:31.769182 28186 ts_tablet_manager.cc:1398] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20240503 08:11:31.770061 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:31.770308 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:31.770516 27495 raft_consensus.cc:2454] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:31.771011 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:31.771246 27235 raft_consensus.cc:2454] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:31.771859 27038 leader_election.cc:302] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:31.772254 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:31.772467 27625 raft_consensus.cc:2454] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:31.773530 28145 raft_consensus.cc:2781] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:31.773639 28145 raft_consensus.cc:482] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:31.773727 28145 raft_consensus.cc:3037] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:11:31.773846 27037 leader_election.cc:341] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:31.776465 28145 raft_consensus.cc:504] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.778002 28186 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:31.779728 28186 tablet_bootstrap.cc:654] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.780841 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:31.781060 27625 raft_consensus.cc:3037] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.781050 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
W20240503 08:11:31.781692 27037 leader_election.cc:341] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:31.782034 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:31.782241 27235 raft_consensus.cc:3037] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.783881 27625 raft_consensus.cc:2454] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:31.785259 27235 raft_consensus.cc:2454] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:31.785357 28145 leader_election.cc:288] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 election: Requested vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:31.785409 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:31.785706 27037 leader_election.cc:302] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80; no voters: fe8fdcdebbff4703a924f83646531115
I20240503 08:11:31.786154 28145 raft_consensus.cc:2781] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:31.786338 28145 raft_consensus.cc:686] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Becoming Leader. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Running, Role: LEADER
I20240503 08:11:31.786713 28145 consensus_queue.cc:237] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.789721 27495 raft_consensus.cc:3037] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.791891 28182 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:31.792586 27495 raft_consensus.cc:2454] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:31.794425 28182 ts_tablet_manager.cc:1367] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.003s
I20240503 08:11:31.795491 28182 raft_consensus.cc:348] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.796803 28182 raft_consensus.cc:374] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.797063 28182 raft_consensus.cc:729] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.797641 28182 consensus_queue.cc:260] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.798723 27428 leader_election.cc:302] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 2de33eb4315b4547889d1e6edae2f2dd, 36fc7a106457453ba91f05b78a5d342b; no voters: f4d3aef6d5624cf4adc3859dbadccb80
I20240503 08:11:31.799587 28182 ts_tablet_manager.cc:1398] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:11:31.799743 28171 raft_consensus.cc:2741] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 08:11:31.801647 28182 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:31.804402 28177 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:31.804649 28177 ts_tablet_manager.cc:1367] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.012s
I20240503 08:11:31.805307 28182 tablet_bootstrap.cc:654] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:31.805595 28178 ts_tablet_manager.cc:1398] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.039s user 0.019s sys 0.001s
I20240503 08:11:31.805459 28177 raft_consensus.cc:348] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.805701 28177 raft_consensus.cc:374] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.805835 28177 raft_consensus.cc:729] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.806093 28177 consensus_queue.cc:260] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.807384 28186 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:31.807665 28186 ts_tablet_manager.cc:1367] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.006s
I20240503 08:11:31.808621 28186 raft_consensus.cc:348] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.808913 28186 raft_consensus.cc:374] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.809010 28186 raft_consensus.cc:729] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.809307 28186 consensus_queue.cc:260] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.809477 26959 catalog_manager.cc:5484] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:31.809857 28186 ts_tablet_manager.cc:1398] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:11:31.810027 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:31.810247 27365 raft_consensus.cc:2454] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:31.811448 28177 ts_tablet_manager.cc:1398] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 08:11:31.811902 28182 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:31.812405 28182 ts_tablet_manager.cc:1367] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240503 08:11:31.813154 28182 raft_consensus.cc:348] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.813423 28182 raft_consensus.cc:374] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:31.813524 28182 raft_consensus.cc:729] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:31.813758 28182 consensus_queue.cc:260] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.814224 28182 ts_tablet_manager.cc:1398] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:11:31.820493 28201 raft_consensus.cc:482] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:31.820657 28201 raft_consensus.cc:504] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.821580 28201 leader_election.cc:288] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:11:31.821997 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b4767b5b4f4fb0bfc73c35a408a5ac" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:31.822091 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b4767b5b4f4fb0bfc73c35a408a5ac" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:31.822201 27365 raft_consensus.cc:2454] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:31.822284 27235 raft_consensus.cc:2454] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:31.822304 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b4767b5b4f4fb0bfc73c35a408a5ac" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:31.822530 27755 raft_consensus.cc:2454] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:31.822736 27558 leader_election.cc:302] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, 36fc7a106457453ba91f05b78a5d342b, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:31.823014 28201 raft_consensus.cc:2781] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:31.823114 28201 raft_consensus.cc:482] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:31.823202 28201 raft_consensus.cc:3037] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.825145 28201 raft_consensus.cc:504] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.826015 28201 leader_election.cc:288] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 election: Requested vote from peers d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:11:31.826086 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b4767b5b4f4fb0bfc73c35a408a5ac" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:31.826273 27755 raft_consensus.cc:3037] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.826396 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b4767b5b4f4fb0bfc73c35a408a5ac" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:31.826592 27235 raft_consensus.cc:3037] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.826599 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b4767b5b4f4fb0bfc73c35a408a5ac" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:31.826769 27365 raft_consensus.cc:3037] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.829144 27235 raft_consensus.cc:2454] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:31.829241 27755 raft_consensus.cc:2454] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:31.829241 27365 raft_consensus.cc:2454] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:31.829666 27558 leader_election.cc:302] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, d8d7c32cb9a241c99853a6639451c4ed, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:31.829964 28201 raft_consensus.cc:2781] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:31.829939 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b4767b5b4f4fb0bfc73c35a408a5ac" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:31.829984 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b4767b5b4f4fb0bfc73c35a408a5ac" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:31.830150 28201 raft_consensus.cc:686] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Becoming Leader. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Running, Role: LEADER
I20240503 08:11:31.830174 27494 raft_consensus.cc:3037] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.830441 28201 consensus_queue.cc:237] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:31.832631 27494 raft_consensus.cc:2454] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:31.833258 26955 catalog_manager.cc:5484] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5). New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:31.869027 28187 raft_consensus.cc:482] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:31.869216 28187 raft_consensus.cc:504] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.870172 28187 leader_election.cc:288] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:31.870536 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4a8224c7c1430a9354e58bd16c7abd" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:31.870615 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4a8224c7c1430a9354e58bd16c7abd" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:31.870636 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4a8224c7c1430a9354e58bd16c7abd" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:31.870828 27625 raft_consensus.cc:2454] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 0.
I20240503 08:11:31.870838 27365 raft_consensus.cc:2454] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 0.
I20240503 08:11:31.870904 27755 raft_consensus.cc:2454] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 0.
I20240503 08:11:31.870958 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4a8224c7c1430a9354e58bd16c7abd" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:31.871316 27168 leader_election.cc:302] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, 36fc7a106457453ba91f05b78a5d342b, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:31.871398 27494 raft_consensus.cc:2454] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 0.
I20240503 08:11:31.871709 28187 raft_consensus.cc:2781] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:31.871846 28187 raft_consensus.cc:482] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:31.871989 28187 raft_consensus.cc:3037] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.873960 28187 raft_consensus.cc:504] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.874912 28187 leader_election.cc:288] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 election: Requested vote from peers 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:31.875087 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4a8224c7c1430a9354e58bd16c7abd" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:31.875159 27365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4a8224c7c1430a9354e58bd16c7abd" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:31.875280 27625 raft_consensus.cc:3037] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.875321 27365 raft_consensus.cc:3037] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.875276 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4a8224c7c1430a9354e58bd16c7abd" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:31.875487 27755 raft_consensus.cc:3037] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.875676 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4a8224c7c1430a9354e58bd16c7abd" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:31.875885 27494 raft_consensus.cc:3037] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:31.877959 27625 raft_consensus.cc:2454] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 1.
I20240503 08:11:31.877990 27365 raft_consensus.cc:2454] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 1.
I20240503 08:11:31.878160 27755 raft_consensus.cc:2454] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 1.
I20240503 08:11:31.878428 27169 leader_election.cc:302] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, 36fc7a106457453ba91f05b78a5d342b, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:31.878727 28187 raft_consensus.cc:2781] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:31.878734 27494 raft_consensus.cc:2454] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 1.
I20240503 08:11:31.879154 28187 raft_consensus.cc:686] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Becoming Leader. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Running, Role: LEADER
I20240503 08:11:31.879886 28187 consensus_queue.cc:237] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:31.883121 26955 catalog_manager.cc:5484] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b reported cstate change: term changed from 0 to 1, leader changed from <none> to 36fc7a106457453ba91f05b78a5d342b (127.24.123.2). New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:32.202677 28145 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:32.222461 28145 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:32.223513 28210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:32.223927 28210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:11:32.227803 28145 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:32.236826 28210 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:32.240667 27365 raft_consensus.cc:3037] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.244618 28219 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:32.264884 28201 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 08:11:32.283921 28201 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:32.288496 28201 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:32.292133 28187 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:32.296685 28245 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:32.422014 28187 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:32.432653 27755 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:32.433504 28208 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:32.439919 27365 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:32.440533 28208 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:32.505616 28276 mvcc.cc:205] Tried to move back new op lower bound from 7023509063395487744 to 7023509061143965696. Current Snapshot: MvccSnapshot[applied={T|T < 7023509063395487744}]
I20240503 08:11:32.523763 28248 mvcc.cc:205] Tried to move back new op lower bound from 7023509063395487744 to 7023509061143965696. Current Snapshot: MvccSnapshot[applied={T|T < 7023509063395487744}]
I20240503 08:11:32.570864 28272 mvcc.cc:205] Tried to move back new op lower bound from 7023509063395487744 to 7023509061143965696. Current Snapshot: MvccSnapshot[applied={T|T < 7023509063395487744 or (T in {7023509063395487744})}]
W20240503 08:11:32.610473 27040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.7:41167: connect: Connection refused (error 111) [suppressed 10 similar messages]
INSERT report
rows total: 2000
time total: 316.006 ms
time per row: 0.158003 ms
I20240503 08:11:32.764343 26959 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47276:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:32.776582 27214 tablet_service.cc:1467] Processing CreateTablet for tablet 43351c2d61024ae0afc96b6cf94d1741 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.777019 27214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43351c2d61024ae0afc96b6cf94d1741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.779199 27733 tablet_service.cc:1467] Processing CreateTablet for tablet 43351c2d61024ae0afc96b6cf94d1741 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.779646 27733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43351c2d61024ae0afc96b6cf94d1741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.779757 27082 tablet_service.cc:1467] Processing CreateTablet for tablet 43351c2d61024ae0afc96b6cf94d1741 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.780054 27084 tablet_service.cc:1467] Processing CreateTablet for tablet a61402b1e1644ab7a0769b74e40e479b (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.780187 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43351c2d61024ae0afc96b6cf94d1741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.780236 27473 tablet_service.cc:1467] Processing CreateTablet for tablet 43351c2d61024ae0afc96b6cf94d1741 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.780407 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61402b1e1644ab7a0769b74e40e479b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.780611 27473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43351c2d61024ae0afc96b6cf94d1741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.785081 27605 tablet_service.cc:1467] Processing CreateTablet for tablet a61402b1e1644ab7a0769b74e40e479b (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.785457 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61402b1e1644ab7a0769b74e40e479b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.786455 27735 tablet_service.cc:1467] Processing CreateTablet for tablet e8c93cddbbf84976baa817ab0d98a811 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.786859 27735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c93cddbbf84976baa817ab0d98a811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.787426 27474 tablet_service.cc:1467] Processing CreateTablet for tablet e8c93cddbbf84976baa817ab0d98a811 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.784159 27734 tablet_service.cc:1467] Processing CreateTablet for tablet a61402b1e1644ab7a0769b74e40e479b (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.787767 27474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c93cddbbf84976baa817ab0d98a811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.787878 27734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61402b1e1644ab7a0769b74e40e479b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.791661 27604 tablet_service.cc:1467] Processing CreateTablet for tablet e8c93cddbbf84976baa817ab0d98a811 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.791999 27604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c93cddbbf84976baa817ab0d98a811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.796855 27344 tablet_service.cc:1467] Processing CreateTablet for tablet a61402b1e1644ab7a0769b74e40e479b (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.797287 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61402b1e1644ab7a0769b74e40e479b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.799576 27215 tablet_service.cc:1467] Processing CreateTablet for tablet a61402b1e1644ab7a0769b74e40e479b (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.799959 27215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61402b1e1644ab7a0769b74e40e479b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.803213 27213 tablet_service.cc:1467] Processing CreateTablet for tablet e8c93cddbbf84976baa817ab0d98a811 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.803372 28301 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:32.803580 27213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c93cddbbf84976baa817ab0d98a811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.804968 28301 tablet_bootstrap.cc:654] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.806366 28302 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:32.807631 27343 tablet_service.cc:1467] Processing CreateTablet for tablet 43351c2d61024ae0afc96b6cf94d1741 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.808039 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43351c2d61024ae0afc96b6cf94d1741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.808126 28302 tablet_bootstrap.cc:654] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.810832 28301 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:32.811148 28301 ts_tablet_manager.cc:1367] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240503 08:11:32.812778 28301 raft_consensus.cc:348] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.812995 28301 raft_consensus.cc:374] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.813081 28301 raft_consensus.cc:729] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.813289 28301 consensus_queue.cc:260] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.818078 28301 ts_tablet_manager.cc:1398] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:11:32.818421 28301 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:32.820214 28301 tablet_bootstrap.cc:654] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.822013 28302 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:32.822170 28302 ts_tablet_manager.cc:1367] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 08:11:32.823029 28302 raft_consensus.cc:348] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.823320 28302 raft_consensus.cc:374] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.823402 28302 raft_consensus.cc:729] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.823632 28302 consensus_queue.cc:260] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.824223 28302 ts_tablet_manager.cc:1398] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:32.824519 28302 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:32.826618 28302 tablet_bootstrap.cc:654] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.834558 28307 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:32.836369 28307 tablet_bootstrap.cc:654] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.844223 28307 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:32.844365 28307 ts_tablet_manager.cc:1367] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240503 08:11:32.845072 28307 raft_consensus.cc:348] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.845324 28307 raft_consensus.cc:374] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.845409 28307 raft_consensus.cc:729] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.845651 28307 consensus_queue.cc:260] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.846323 28307 ts_tablet_manager.cc:1398] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:32.846558 28307 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:32.848016 28307 tablet_bootstrap.cc:654] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.863850 27082 tablet_service.cc:1467] Processing CreateTablet for tablet e8c93cddbbf84976baa817ab0d98a811 (DEFAULT_TABLE table=rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.864470 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c93cddbbf84976baa817ab0d98a811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.887701 28309 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:32.889585 28309 tablet_bootstrap.cc:654] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.891045 28310 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:32.891793 28312 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:32.893256 28310 tablet_bootstrap.cc:654] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.893591 28312 tablet_bootstrap.cc:654] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.897277 28307 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:32.897503 28307 ts_tablet_manager.cc:1367] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.026s
I20240503 08:11:32.897657 28309 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:32.898335 28307 raft_consensus.cc:348] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.898607 28309 ts_tablet_manager.cc:1367] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:11:32.898706 28307 raft_consensus.cc:374] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.898823 28307 raft_consensus.cc:729] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.898854 28302 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:32.899052 28302 ts_tablet_manager.cc:1367] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.025s
I20240503 08:11:32.899385 28309 raft_consensus.cc:348] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.899389 28307 consensus_queue.cc:260] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.899616 28309 raft_consensus.cc:374] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.899708 28309 raft_consensus.cc:729] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.899792 28302 raft_consensus.cc:348] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.900040 28302 raft_consensus.cc:374] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.899952 28309 consensus_queue.cc:260] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.900100 28302 raft_consensus.cc:729] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.900432 28302 consensus_queue.cc:260] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.901392 28309 ts_tablet_manager.cc:1398] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:32.901647 28309 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:32.902185 28301 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:32.902345 28301 ts_tablet_manager.cc:1367] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.084s user 0.014s sys 0.024s
I20240503 08:11:32.903132 28301 raft_consensus.cc:348] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.903386 28301 raft_consensus.cc:374] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.903457 28301 raft_consensus.cc:729] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.903726 28301 consensus_queue.cc:260] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.903928 28310 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:32.904076 28310 ts_tablet_manager.cc:1367] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 08:11:32.904274 28307 ts_tablet_manager.cc:1398] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:11:32.904875 28301 ts_tablet_manager.cc:1398] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:32.905153 28301 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:32.906512 28312 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:32.906410 28310 raft_consensus.cc:348] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.906922 28312 ts_tablet_manager.cc:1367] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 08:11:32.907424 28301 tablet_bootstrap.cc:654] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.907758 28312 raft_consensus.cc:348] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.908128 28312 raft_consensus.cc:374] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.906638 28310 raft_consensus.cc:374] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.908272 28312 raft_consensus.cc:729] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.908356 28310 raft_consensus.cc:729] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.908648 28302 ts_tablet_manager.cc:1398] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 08:11:32.908627 28310 consensus_queue.cc:260] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.908607 28312 consensus_queue.cc:260] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.909271 28312 ts_tablet_manager.cc:1398] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:32.909441 28302 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:32.909821 28310 ts_tablet_manager.cc:1398] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:11:32.910066 28310 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:32.910432 28312 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:32.910939 28302 tablet_bootstrap.cc:654] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.912021 28312 tablet_bootstrap.cc:654] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.915875 28309 tablet_bootstrap.cc:654] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.916833 28310 tablet_bootstrap.cc:654] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.918372 28302 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:32.918517 28302 ts_tablet_manager.cc:1367] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20240503 08:11:32.919245 28302 raft_consensus.cc:348] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.919508 28302 raft_consensus.cc:374] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.919596 28302 raft_consensus.cc:729] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.921098 28301 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:32.921255 28301 ts_tablet_manager.cc:1367] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 08:11:32.922313 28301 raft_consensus.cc:348] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.922564 28301 raft_consensus.cc:374] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.922648 28301 raft_consensus.cc:729] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.922931 28301 consensus_queue.cc:260] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.924105 28301 ts_tablet_manager.cc:1398] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:32.924461 28302 consensus_queue.cc:260] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.926297 28302 ts_tablet_manager.cc:1398] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 08:11:32.930583 28309 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:32.930815 28309 ts_tablet_manager.cc:1367] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20240503 08:11:32.930975 28310 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:32.931140 28310 ts_tablet_manager.cc:1367] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.006s
I20240503 08:11:32.931223 28312 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:32.931473 28312 ts_tablet_manager.cc:1367] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20240503 08:11:32.931908 28309 raft_consensus.cc:348] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.931967 28310 raft_consensus.cc:348] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.932195 28310 raft_consensus.cc:374] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.932271 28310 raft_consensus.cc:729] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.932303 28309 raft_consensus.cc:374] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.932235 28312 raft_consensus.cc:348] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.932513 28312 raft_consensus.cc:374] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.932512 28309 raft_consensus.cc:729] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.932591 28312 raft_consensus.cc:729] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.932543 28310 consensus_queue.cc:260] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.932847 28312 consensus_queue.cc:260] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.933507 28312 ts_tablet_manager.cc:1398] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:32.932947 28309 consensus_queue.cc:260] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:32.934103 28309 ts_tablet_manager.cc:1398] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:32.934239 28310 ts_tablet_manager.cc:1398] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:11:32.934399 28309 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:32.936019 28309 tablet_bootstrap.cc:654] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.940162 28309 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:32.940302 28309 ts_tablet_manager.cc:1367] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 08:11:32.941040 28309 raft_consensus.cc:348] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.941260 28309 raft_consensus.cc:374] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.941326 28309 raft_consensus.cc:729] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.941524 28309 consensus_queue.cc:260] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.942013 28309 ts_tablet_manager.cc:1398] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:32.972082 28171 raft_consensus.cc:482] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:32.972268 28171 raft_consensus.cc:504] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.973217 28171 leader_election.cc:288] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:11:32.973505 28201 raft_consensus.cc:482] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:32.973659 27363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43351c2d61024ae0afc96b6cf94d1741" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:32.973677 28201 raft_consensus.cc:504] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.973875 27363 raft_consensus.cc:2454] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:32.974005 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43351c2d61024ae0afc96b6cf94d1741" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:32.974216 27234 raft_consensus.cc:2454] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:32.974434 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43351c2d61024ae0afc96b6cf94d1741" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:32.974582 28201 leader_election.cc:288] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:32.974664 27753 raft_consensus.cc:2454] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:32.975558 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43351c2d61024ae0afc96b6cf94d1741" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:32.975680 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61402b1e1644ab7a0769b74e40e479b" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:32.975776 27104 raft_consensus.cc:2454] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:32.975878 27103 raft_consensus.cc:2454] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:32.976450 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61402b1e1644ab7a0769b74e40e479b" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:32.976547 27431 leader_election.cc:302] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, d8d7c32cb9a241c99853a6639451c4ed, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:32.976711 27753 raft_consensus.cc:2454] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:32.977140 27561 leader_election.cc:302] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, d8d7c32cb9a241c99853a6639451c4ed, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:32.977262 27363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61402b1e1644ab7a0769b74e40e479b" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:32.977438 27363 raft_consensus.cc:2454] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:32.977753 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61402b1e1644ab7a0769b74e40e479b" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:32.977926 28171 raft_consensus.cc:2781] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:32.977965 27234 raft_consensus.cc:2454] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:32.978030 28171 raft_consensus.cc:482] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:32.978107 28171 raft_consensus.cc:3037] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.978610 28201 raft_consensus.cc:2781] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:32.978729 28201 raft_consensus.cc:482] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:32.978804 28201 raft_consensus.cc:3037] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.980814 28171 raft_consensus.cc:504] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:32.981424 28201 raft_consensus.cc:504] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:32.982214 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43351c2d61024ae0afc96b6cf94d1741" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:32.982388 27103 raft_consensus.cc:3037] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.982715 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61402b1e1644ab7a0769b74e40e479b" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:32.983037 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43351c2d61024ae0afc96b6cf94d1741" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:32.983222 27234 raft_consensus.cc:3037] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.985085 27103 raft_consensus.cc:2454] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:32.985826 27234 raft_consensus.cc:2454] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:32.986272 28171 leader_election.cc:288] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 election: Requested vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:11:32.986663 27428 leader_election.cc:302] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:32.986923 27363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43351c2d61024ae0afc96b6cf94d1741" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:32.987102 27363 raft_consensus.cc:3037] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.987849 27753 raft_consensus.cc:3037] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.989830 27363 raft_consensus.cc:2454] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:32.990440 27753 raft_consensus.cc:2454] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:32.990664 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43351c2d61024ae0afc96b6cf94d1741" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:32.990830 27755 raft_consensus.cc:3037] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.993436 27755 raft_consensus.cc:2454] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:32.995815 28201 leader_election.cc:288] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 election: Requested vote from peers d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:32.995857 28171 raft_consensus.cc:2781] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:32.996423 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61402b1e1644ab7a0769b74e40e479b" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:32.996654 27103 raft_consensus.cc:3037] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.996789 27363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61402b1e1644ab7a0769b74e40e479b" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:32.997104 27363 raft_consensus.cc:3037] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.999068 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61402b1e1644ab7a0769b74e40e479b" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:32.999261 27234 raft_consensus.cc:3037] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.000146 27363 raft_consensus.cc:2454] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:33.000726 28171 raft_consensus.cc:686] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Becoming Leader. State: Replica: 19fb518a6ede476f809c743054941e97, State: Running, Role: LEADER
I20240503 08:11:33.000699 27558 leader_election.cc:302] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, d8d7c32cb9a241c99853a6639451c4ed, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:33.001024 28201 raft_consensus.cc:2781] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:33.001159 28201 raft_consensus.cc:686] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Becoming Leader. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Running, Role: LEADER
I20240503 08:11:33.001042 28171 consensus_queue.cc:237] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:33.001569 28201 consensus_queue.cc:237] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } }
I20240503 08:11:33.003487 27234 raft_consensus.cc:2454] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:33.004276 27103 raft_consensus.cc:2454] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:33.004956 26958 catalog_manager.cc:5484] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5). New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:33.006860 26959 catalog_manager.cc:5484] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19fb518a6ede476f809c743054941e97 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:33.132658 27040 consensus_peers.cc:485] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.7:41167: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:11:33.152537 28172 raft_consensus.cc:482] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:33.152735 28172 raft_consensus.cc:504] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:33.153942 28172 leader_election.cc:288] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465)
I20240503 08:11:33.154196 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:33.154282 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:33.154428 27234 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:33.154536 27494 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:33.154502 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:33.154517 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:33.154731 27624 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:33.154803 27755 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
I20240503 08:11:33.155143 27038 leader_election.cc:302] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:33.155470 28172 raft_consensus.cc:2781] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:33.155581 28172 raft_consensus.cc:482] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:33.155675 28172 raft_consensus.cc:3037] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.158103 28172 raft_consensus.cc:504] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:33.159145 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:33.159178 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:33.159346 27494 raft_consensus.cc:3037] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.159368 27234 raft_consensus.cc:3037] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.159492 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:33.159575 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:33.159654 27624 raft_consensus.cc:3037] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.159745 27755 raft_consensus.cc:3037] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.161981 27234 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:33.161996 27494 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:33.162247 27755 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:33.158915 28172 leader_election.cc:288] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 election: Requested vote from peers 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465)
I20240503 08:11:33.162639 27624 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 1.
I20240503 08:11:33.162775 27037 leader_election.cc:302] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:33.163100 28172 raft_consensus.cc:2781] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:33.163254 28172 raft_consensus.cc:686] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Becoming Leader. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Running, Role: LEADER
I20240503 08:11:33.163515 28172 consensus_queue.cc:237] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:33.166596 26959 catalog_manager.cc:5484] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:33.379880 28201 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.448422 28171 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.454496 28264 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.460500 28201 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.472978 28171 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.493163 28264 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.502077 28322 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.502494 28171 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.572957 28172 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:33.576241 28328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:33.576608 28328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:33.584332 28328 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:11:33.661427 28172 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.672425 28172 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:33.681087 28145 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:33.982836 28329 log.cc:924] Time spent T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
INSERT report
rows total: 2000
time total: 372.26 ms
time per row: 0.18613 ms
I20240503 08:11:34.100760 26957 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47276:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:34.109320 27343 tablet_service.cc:1467] Processing CreateTablet for tablet 921b4706d2bc4693ae5a0a10b310d5d4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.109385 27604 tablet_service.cc:1467] Processing CreateTablet for tablet 921b4706d2bc4693ae5a0a10b310d5d4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.109784 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921b4706d2bc4693ae5a0a10b310d5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.109784 27604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921b4706d2bc4693ae5a0a10b310d5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.109789 27474 tablet_service.cc:1467] Processing CreateTablet for tablet 921b4706d2bc4693ae5a0a10b310d5d4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.110193 27474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921b4706d2bc4693ae5a0a10b310d5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.114919 28422 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:34.115581 27603 tablet_service.cc:1467] Processing CreateTablet for tablet 89392b248b7746d5bf8b95ac9d662e43 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.115738 27605 tablet_service.cc:1467] Processing CreateTablet for tablet f09f0689f0714d8c8f45e24bd9adb533 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.115994 27603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89392b248b7746d5bf8b95ac9d662e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.116151 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09f0689f0714d8c8f45e24bd9adb533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.116582 28422 tablet_bootstrap.cc:654] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.119292 27473 tablet_service.cc:1467] Processing CreateTablet for tablet f09f0689f0714d8c8f45e24bd9adb533 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.119679 27473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09f0689f0714d8c8f45e24bd9adb533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.121907 28423 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:34.122938 27475 tablet_service.cc:1467] Processing CreateTablet for tablet 89392b248b7746d5bf8b95ac9d662e43 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.123196 27083 tablet_service.cc:1467] Processing CreateTablet for tablet 89392b248b7746d5bf8b95ac9d662e43 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.123302 27475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89392b248b7746d5bf8b95ac9d662e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.123589 27083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89392b248b7746d5bf8b95ac9d662e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.123663 28425 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:34.126214 28423 tablet_bootstrap.cc:654] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.127259 28425 tablet_bootstrap.cc:654] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.127298 27082 tablet_service.cc:1467] Processing CreateTablet for tablet 921b4706d2bc4693ae5a0a10b310d5d4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.127328 27734 tablet_service.cc:1467] Processing CreateTablet for tablet 921b4706d2bc4693ae5a0a10b310d5d4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.127614 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921b4706d2bc4693ae5a0a10b310d5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.127758 27734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921b4706d2bc4693ae5a0a10b310d5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.131292 27084 tablet_service.cc:1467] Processing CreateTablet for tablet f09f0689f0714d8c8f45e24bd9adb533 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.131688 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09f0689f0714d8c8f45e24bd9adb533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.133002 27733 tablet_service.cc:1467] Processing CreateTablet for tablet f09f0689f0714d8c8f45e24bd9adb533 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.133394 27733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09f0689f0714d8c8f45e24bd9adb533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.140767 27343 tablet_service.cc:1467] Processing CreateTablet for tablet f09f0689f0714d8c8f45e24bd9adb533 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.141191 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09f0689f0714d8c8f45e24bd9adb533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.143276 27344 tablet_service.cc:1467] Processing CreateTablet for tablet 89392b248b7746d5bf8b95ac9d662e43 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.143659 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89392b248b7746d5bf8b95ac9d662e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.147097 28428 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:34.147359 27215 tablet_service.cc:1467] Processing CreateTablet for tablet 89392b248b7746d5bf8b95ac9d662e43 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.147787 27215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89392b248b7746d5bf8b95ac9d662e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.149281 28428 tablet_bootstrap.cc:654] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.151297 28425 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:34.151530 28425 ts_tablet_manager.cc:1367] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.012s
I20240503 08:11:34.152364 28425 raft_consensus.cc:348] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.152747 28425 raft_consensus.cc:374] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.152884 28425 raft_consensus.cc:729] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.153036 28422 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:34.153219 28422 ts_tablet_manager.cc:1367] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.038s user 0.016s sys 0.021s
I20240503 08:11:34.153249 28425 consensus_queue.cc:260] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.154073 28425 ts_tablet_manager.cc:1398] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:34.153954 28422 raft_consensus.cc:348] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.154172 28422 raft_consensus.cc:374] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.154314 28422 raft_consensus.cc:729] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.154561 28425 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:34.154536 28422 consensus_queue.cc:260] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.155198 28422 ts_tablet_manager.cc:1398] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:34.158140 28428 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:34.158277 28428 ts_tablet_manager.cc:1367] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:11:34.159026 28428 raft_consensus.cc:348] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.159250 28428 raft_consensus.cc:374] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.159338 28428 raft_consensus.cc:729] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.159543 28428 consensus_queue.cc:260] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.160081 28428 ts_tablet_manager.cc:1398] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:34.160385 28428 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:34.161867 28428 tablet_bootstrap.cc:654] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.161897 28425 tablet_bootstrap.cc:654] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.170681 28430 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:34.172358 28430 tablet_bootstrap.cc:654] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.173629 28422 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:34.174856 28428 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:34.175040 28428 ts_tablet_manager.cc:1367] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20240503 08:11:34.175172 28422 tablet_bootstrap.cc:654] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.175737 28433 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:34.175773 28428 raft_consensus.cc:348] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.176025 28428 raft_consensus.cc:374] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.176095 28428 raft_consensus.cc:729] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.176330 28428 consensus_queue.cc:260] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.176851 28428 ts_tablet_manager.cc:1398] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:34.177115 28428 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:34.178575 28428 tablet_bootstrap.cc:654] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.178961 28423 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:34.179195 28423 ts_tablet_manager.cc:1367] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.011s
I20240503 08:11:34.180156 28423 raft_consensus.cc:348] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.180496 28423 raft_consensus.cc:374] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.180619 28423 raft_consensus.cc:729] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.180938 28423 consensus_queue.cc:260] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.182852 28422 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:34.183063 28422 ts_tablet_manager.cc:1367] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:11:34.183869 28422 raft_consensus.cc:348] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.184146 28422 raft_consensus.cc:374] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.184230 28422 raft_consensus.cc:729] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.184463 28422 consensus_queue.cc:260] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.185035 28422 ts_tablet_manager.cc:1398] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:34.185354 28422 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:34.187391 28422 tablet_bootstrap.cc:654] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.188555 28433 tablet_bootstrap.cc:654] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.194011 28425 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:34.194258 28425 ts_tablet_manager.cc:1367] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.000s
I20240503 08:11:34.194582 28375 raft_consensus.cc:482] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:34.194759 28375 raft_consensus.cc:504] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.195854 28375 leader_election.cc:288] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:34.195969 28428 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:34.196113 28428 ts_tablet_manager.cc:1367] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20240503 08:11:34.196467 28423 ts_tablet_manager.cc:1398] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.017s user 0.003s sys 0.001s
I20240503 08:11:34.196575 28430 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:34.196731 28430 ts_tablet_manager.cc:1367] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.018s
I20240503 08:11:34.197021 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:34.197108 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:34.197448 28423 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:34.196863 28428 raft_consensus.cc:348] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.197485 28430 raft_consensus.cc:348] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.197712 28428 raft_consensus.cc:374] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.197736 28430 raft_consensus.cc:374] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.197814 28428 raft_consensus.cc:729] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.197832 28430 raft_consensus.cc:729] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.197786 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:34.198041 28428 consensus_queue.cc:260] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.198042 28430 consensus_queue.cc:260] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.197127 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:34.198421 27624 raft_consensus.cc:2454] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d3aef6d5624cf4adc3859dbadccb80 in term 0.
W20240503 08:11:34.198541 27040 leader_election.cc:341] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:34.198657 28428 ts_tablet_manager.cc:1398] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240503 08:11:34.198781 27038 leader_election.cc:341] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:11:34.199080 27037 leader_election.cc:341] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:34.199221 27037 leader_election.cc:302] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: f4d3aef6d5624cf4adc3859dbadccb80; no voters: 19fb518a6ede476f809c743054941e97, d8d7c32cb9a241c99853a6639451c4ed, fe8fdcdebbff4703a924f83646531115
I20240503 08:11:34.199764 28375 raft_consensus.cc:2726] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:34.199534 28425 raft_consensus.cc:348] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.203208 28425 raft_consensus.cc:374] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.203311 28425 raft_consensus.cc:729] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.203567 28425 consensus_queue.cc:260] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.204160 28425 ts_tablet_manager.cc:1398] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:11:34.204425 28425 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:34.205107 28422 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:34.205310 28422 ts_tablet_manager.cc:1367] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240503 08:11:34.206090 28425 tablet_bootstrap.cc:654] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.206102 28422 raft_consensus.cc:348] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.206452 28422 raft_consensus.cc:374] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.206513 28423 tablet_bootstrap.cc:654] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.206619 28422 raft_consensus.cc:729] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.206970 28422 consensus_queue.cc:260] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.207566 28422 ts_tablet_manager.cc:1398] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:34.208294 28433 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:34.208631 28433 ts_tablet_manager.cc:1367] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.001s
I20240503 08:11:34.209432 28433 raft_consensus.cc:348] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.209673 28433 raft_consensus.cc:374] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.209761 28433 raft_consensus.cc:729] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.210716 28433 consensus_queue.cc:260] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.213332 28433 ts_tablet_manager.cc:1398] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:11:34.215866 28430 ts_tablet_manager.cc:1398] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.019s user 0.008s sys 0.009s
I20240503 08:11:34.216224 28430 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:34.218037 28430 tablet_bootstrap.cc:654] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.221345 28423 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:34.221542 28423 ts_tablet_manager.cc:1367] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.010s
I20240503 08:11:34.222311 28423 raft_consensus.cc:348] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.222589 28423 raft_consensus.cc:374] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.222659 28423 raft_consensus.cc:729] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.222718 28425 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:34.222898 28425 ts_tablet_manager.cc:1367] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 08:11:34.222960 28423 consensus_queue.cc:260] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.223657 28423 ts_tablet_manager.cc:1398] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:34.223701 28425 raft_consensus.cc:348] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.223981 28425 raft_consensus.cc:374] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.224066 28425 raft_consensus.cc:729] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.224314 28425 consensus_queue.cc:260] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.226778 28425 ts_tablet_manager.cc:1398] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:34.226418 28423 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:34.228226 28430 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:34.229223 28430 ts_tablet_manager.cc:1367] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20240503 08:11:34.229619 28423 tablet_bootstrap.cc:654] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.230052 28430 raft_consensus.cc:348] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.230275 28430 raft_consensus.cc:374] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.230350 28430 raft_consensus.cc:729] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.230558 28430 consensus_queue.cc:260] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.231087 28430 ts_tablet_manager.cc:1398] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:34.238907 28410 raft_consensus.cc:482] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:34.239039 28423 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:34.239207 28423 ts_tablet_manager.cc:1367] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240503 08:11:34.239077 28410 raft_consensus.cc:504] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.240026 28410 leader_election.cc:288] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:34.240008 28423 raft_consensus.cc:348] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.240267 28423 raft_consensus.cc:374] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.240681 28423 raft_consensus.cc:729] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.240991 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:34.240989 28423 consensus_queue.cc:260] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.241210 27753 raft_consensus.cc:2454] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:34.241534 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:34.241731 27103 raft_consensus.cc:2454] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:34.242101 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:34.242164 28423 ts_tablet_manager.cc:1398] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:34.242306 27624 raft_consensus.cc:2454] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:34.242997 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:34.243166 27431 leader_election.cc:302] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 2de33eb4315b4547889d1e6edae2f2dd, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:34.243227 27364 raft_consensus.cc:2454] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 0.
I20240503 08:11:34.243989 28410 raft_consensus.cc:2781] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:34.244107 28410 raft_consensus.cc:482] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:34.244230 28410 raft_consensus.cc:3037] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.247082 28410 raft_consensus.cc:504] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.248294 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:34.248484 27103 raft_consensus.cc:3037] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.248479 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:34.248668 27364 raft_consensus.cc:3037] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.248914 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:34.249101 27753 raft_consensus.cc:3037] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.250025 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:34.250226 27624 raft_consensus.cc:3037] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.251263 27364 raft_consensus.cc:2454] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:34.251284 27103 raft_consensus.cc:2454] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:34.251767 27753 raft_consensus.cc:2454] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:34.252352 27431 leader_election.cc:302] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:34.252657 28171 raft_consensus.cc:2781] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:34.252790 28410 leader_election.cc:288] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 1 election: Requested vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:34.252816 27624 raft_consensus.cc:2454] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 1.
I20240503 08:11:34.252805 28171 raft_consensus.cc:686] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Becoming Leader. State: Replica: 19fb518a6ede476f809c743054941e97, State: Running, Role: LEADER
I20240503 08:11:34.253163 28171 consensus_queue.cc:237] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:34.256065 26958 catalog_manager.cc:5484] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19fb518a6ede476f809c743054941e97 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:34.302994 28437 raft_consensus.cc:482] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:34.303164 28437 raft_consensus.cc:504] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.304961 28437 leader_election.cc:288] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465)
I20240503 08:11:34.317528 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:34.317783 27624 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:34.321758 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:34.322000 27103 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:34.322365 27301 leader_election.cc:302] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:34.322769 28437 raft_consensus.cc:2781] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:34.322907 28437 raft_consensus.cc:482] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:34.322995 28437 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.323177 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:34.323436 27753 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:34.325956 28437 raft_consensus.cc:504] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.326889 28437 leader_election.cc:288] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 election: Requested vote from peers 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465)
I20240503 08:11:34.327069 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:34.327248 27624 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.327512 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:34.327628 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:34.327705 27103 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.327849 27494 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.329309 27624 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:34.329649 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:34.330250 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:34.330332 27103 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:34.330410 27494 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:34.330425 27753 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.330741 27301 leader_election.cc:302] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:34.331050 28437 raft_consensus.cc:2781] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:34.331496 28437 raft_consensus.cc:686] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Becoming Leader. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Running, Role: LEADER
I20240503 08:11:34.331964 28437 consensus_queue.cc:237] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } }
I20240503 08:11:34.332751 27753 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:34.335562 26958 catalog_manager.cc:5484] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe8fdcdebbff4703a924f83646531115 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:34.476778 28264 raft_consensus.cc:482] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:34.476972 28264 raft_consensus.cc:504] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.477782 28264 leader_election.cc:288] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:11:34.478107 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89392b248b7746d5bf8b95ac9d662e43" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:34.478168 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89392b248b7746d5bf8b95ac9d662e43" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:34.478253 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89392b248b7746d5bf8b95ac9d662e43" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115" is_pre_election: true
I20240503 08:11:34.478360 27494 raft_consensus.cc:2454] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:34.478412 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89392b248b7746d5bf8b95ac9d662e43" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:34.478506 27364 raft_consensus.cc:2454] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:34.478483 27103 raft_consensus.cc:2454] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:34.478626 27234 raft_consensus.cc:2454] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 0.
I20240503 08:11:34.478866 27558 leader_election.cc:302] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 2de33eb4315b4547889d1e6edae2f2dd, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:34.479194 28264 raft_consensus.cc:2781] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:34.479349 28264 raft_consensus.cc:482] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:34.479446 28264 raft_consensus.cc:3037] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.481490 28264 raft_consensus.cc:504] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.482276 28264 leader_election.cc:288] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 election: Requested vote from peers f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:11:34.482398 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89392b248b7746d5bf8b95ac9d662e43" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:34.482421 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89392b248b7746d5bf8b95ac9d662e43" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:34.482580 27234 raft_consensus.cc:3037] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.482606 27103 raft_consensus.cc:3037] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.482751 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89392b248b7746d5bf8b95ac9d662e43" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:34.482996 27494 raft_consensus.cc:3037] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.482978 27364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89392b248b7746d5bf8b95ac9d662e43" candidate_uuid: "2de33eb4315b4547889d1e6edae2f2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:34.483160 27364 raft_consensus.cc:3037] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.485256 27234 raft_consensus.cc:2454] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:34.485527 27103 raft_consensus.cc:2454] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:34.485579 27494 raft_consensus.cc:2454] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:34.485713 27364 raft_consensus.cc:2454] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2de33eb4315b4547889d1e6edae2f2dd in term 1.
I20240503 08:11:34.485823 27561 leader_election.cc:302] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:34.486182 28264 raft_consensus.cc:2781] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:34.486526 28264 raft_consensus.cc:686] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Becoming Leader. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Running, Role: LEADER
I20240503 08:11:34.486793 28264 consensus_queue.cc:237] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:34.489742 26958 catalog_manager.cc:5484] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5). New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:34.636441 28322 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.722455 28437 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.723116 28322 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.732895 28322 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.749452 28437 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.751080 28322 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.770514 28437 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.778352 28437 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:34.879535 28458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:34.879838 28458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:34.887789 28458 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:11:34.889133 28264 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.893925 28456 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.897729 28456 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:34.899755 28456 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 281.562 ms
time per row: 0.140781 ms
W20240503 08:11:35.737243 27040 consensus_peers.cc:485] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.7:41167: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
-----------------------------------------------------
Suppressions used:
count bytes template
14 3624 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:36.205096 26958 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47276:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:36.215847 27084 tablet_service.cc:1467] Processing CreateTablet for tablet ccb3a8dde7ec4d318b0961c8bad8032c (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.216053 27733 tablet_service.cc:1467] Processing CreateTablet for tablet ccb3a8dde7ec4d318b0961c8bad8032c (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.216084 27473 tablet_service.cc:1467] Processing CreateTablet for tablet ccb3a8dde7ec4d318b0961c8bad8032c (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.216291 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb3a8dde7ec4d318b0961c8bad8032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.216298 27734 tablet_service.cc:1467] Processing CreateTablet for tablet 8e17a88a118b4ce591a266fd62b3d7a7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.216446 27473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb3a8dde7ec4d318b0961c8bad8032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.216467 27733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb3a8dde7ec4d318b0961c8bad8032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.216676 27734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e17a88a118b4ce591a266fd62b3d7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.215265 27215 tablet_service.cc:1467] Processing CreateTablet for tablet ccb3a8dde7ec4d318b0961c8bad8032c (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.222864 27215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb3a8dde7ec4d318b0961c8bad8032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.223102 27344 tablet_service.cc:1467] Processing CreateTablet for tablet ccb3a8dde7ec4d318b0961c8bad8032c (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.223614 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb3a8dde7ec4d318b0961c8bad8032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.223654 27603 tablet_service.cc:1467] Processing CreateTablet for tablet 8e17a88a118b4ce591a266fd62b3d7a7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.224013 27603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e17a88a118b4ce591a266fd62b3d7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.227877 27475 tablet_service.cc:1467] Processing CreateTablet for tablet 4c364e9ddad74e8e882df9080d9d5372 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.228009 27214 tablet_service.cc:1467] Processing CreateTablet for tablet 8e17a88a118b4ce591a266fd62b3d7a7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.228252 27475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c364e9ddad74e8e882df9080d9d5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.228329 27214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e17a88a118b4ce591a266fd62b3d7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.228775 27343 tablet_service.cc:1467] Processing CreateTablet for tablet 8e17a88a118b4ce591a266fd62b3d7a7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.229146 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e17a88a118b4ce591a266fd62b3d7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.231230 27213 tablet_service.cc:1467] Processing CreateTablet for tablet 4c364e9ddad74e8e882df9080d9d5372 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.231547 27213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c364e9ddad74e8e882df9080d9d5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.231519 27083 tablet_service.cc:1467] Processing CreateTablet for tablet 8e17a88a118b4ce591a266fd62b3d7a7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.231870 27083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e17a88a118b4ce591a266fd62b3d7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.236236 27605 tablet_service.cc:1467] Processing CreateTablet for tablet 4c364e9ddad74e8e882df9080d9d5372 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.236479 27735 tablet_service.cc:1467] Processing CreateTablet for tablet 4c364e9ddad74e8e882df9080d9d5372 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.236685 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c364e9ddad74e8e882df9080d9d5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.236855 27735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c364e9ddad74e8e882df9080d9d5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.239547 27082 tablet_service.cc:1467] Processing CreateTablet for tablet 4c364e9ddad74e8e882df9080d9d5372 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:36.239936 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c364e9ddad74e8e882df9080d9d5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:36.262703 28547 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:36.263269 28550 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:36.263553 28551 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:36.264914 28550 tablet_bootstrap.cc:654] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.265730 28549 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:36.266100 28547 tablet_bootstrap.cc:654] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.266564 28548 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:36.266727 28551 tablet_bootstrap.cc:654] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.267223 28549 tablet_bootstrap.cc:654] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.268147 28548 tablet_bootstrap.cc:654] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.269343 28546 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:36.270731 28548 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:36.270839 28546 tablet_bootstrap.cc:654] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.270944 28548 ts_tablet_manager.cc:1367] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:36.271804 28548 raft_consensus.cc:348] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.272131 28548 raft_consensus.cc:374] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.272255 28548 raft_consensus.cc:729] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.272558 28548 consensus_queue.cc:260] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.274008 28548 ts_tablet_manager.cc:1398] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:36.274691 28548 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115: Bootstrap starting.
I20240503 08:11:36.276473 28548 tablet_bootstrap.cc:654] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.279199 28549 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:36.279354 28549 ts_tablet_manager.cc:1367] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240503 08:11:36.281240 28551 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:36.281574 28549 raft_consensus.cc:348] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.281857 28549 raft_consensus.cc:374] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.281965 28549 raft_consensus.cc:729] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.282037 28547 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:36.282194 28547 ts_tablet_manager.cc:1367] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240503 08:11:36.282980 28550 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:36.282785 28549 consensus_queue.cc:260] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.283200 28550 ts_tablet_manager.cc:1367] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240503 08:11:36.283049 28547 raft_consensus.cc:348] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.283290 28547 raft_consensus.cc:374] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.283401 28547 raft_consensus.cc:729] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.283680 28547 consensus_queue.cc:260] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.284153 28546 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:36.284128 28550 raft_consensus.cc:348] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.284381 28546 ts_tablet_manager.cc:1367] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 08:11:36.284431 28550 raft_consensus.cc:374] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.284595 28550 raft_consensus.cc:729] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.284849 28550 consensus_queue.cc:260] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.285249 28547 ts_tablet_manager.cc:1398] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 08:11:36.285568 28547 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:36.285578 28550 ts_tablet_manager.cc:1398] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:36.285946 28550 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:36.286375 28551 ts_tablet_manager.cc:1367] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.008s
I20240503 08:11:36.287159 28551 raft_consensus.cc:348] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.287400 28551 raft_consensus.cc:374] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.287508 28551 raft_consensus.cc:729] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.287788 28550 tablet_bootstrap.cc:654] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.288426 28551 consensus_queue.cc:260] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.289794 28551 ts_tablet_manager.cc:1398] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:36.290071 28551 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd: Bootstrap starting.
I20240503 08:11:36.290459 28547 tablet_bootstrap.cc:654] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.291592 28551 tablet_bootstrap.cc:654] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.295681 28550 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:36.295730 28548 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115: No bootstrap required, opened a new log
I20240503 08:11:36.295893 28548 ts_tablet_manager.cc:1367] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240503 08:11:36.295904 28550 ts_tablet_manager.cc:1367] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 08:11:36.296630 28548 raft_consensus.cc:348] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.296905 28548 raft_consensus.cc:374] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.296983 28548 raft_consensus.cc:729] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.297252 28548 consensus_queue.cc:260] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.297588 28550 raft_consensus.cc:348] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.297890 28550 raft_consensus.cc:374] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.297999 28550 raft_consensus.cc:729] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.298280 28550 consensus_queue.cc:260] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.299477 28550 ts_tablet_manager.cc:1398] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:36.299787 28550 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:36.301858 28550 tablet_bootstrap.cc:654] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.301641 28548 ts_tablet_manager.cc:1398] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:11:36.304772 28547 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:36.304949 28547 ts_tablet_manager.cc:1367] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20240503 08:11:36.305433 28546 raft_consensus.cc:348] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.305723 28546 raft_consensus.cc:374] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.305820 28546 raft_consensus.cc:729] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.305862 28547 raft_consensus.cc:348] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.306125 28547 raft_consensus.cc:374] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.306198 28547 raft_consensus.cc:729] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.306092 28546 consensus_queue.cc:260] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.306468 28547 consensus_queue.cc:260] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.307225 28547 ts_tablet_manager.cc:1398] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:11:36.307518 28547 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
I20240503 08:11:36.309094 28547 tablet_bootstrap.cc:654] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.307451 28546 ts_tablet_manager.cc:1398] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 08:11:36.313220 28546 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97: Bootstrap starting.
I20240503 08:11:36.314790 28546 tablet_bootstrap.cc:654] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.316008 28549 ts_tablet_manager.cc:1398] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240503 08:11:36.316409 28549 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:36.316824 28547 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: No bootstrap required, opened a new log
I20240503 08:11:36.316946 28547 ts_tablet_manager.cc:1367] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240503 08:11:36.317996 28549 tablet_bootstrap.cc:654] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.318687 28547 raft_consensus.cc:348] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.318920 28547 raft_consensus.cc:374] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.319011 28547 raft_consensus.cc:729] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.319242 28547 consensus_queue.cc:260] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.319643 28546 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97: No bootstrap required, opened a new log
I20240503 08:11:36.319802 28546 ts_tablet_manager.cc:1367] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:11:36.319799 28547 ts_tablet_manager.cc:1398] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:36.320457 28546 raft_consensus.cc:348] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.320670 28546 raft_consensus.cc:374] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.320739 28546 raft_consensus.cc:729] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb518a6ede476f809c743054941e97, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.320982 28546 consensus_queue.cc:260] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.321422 28551 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd: No bootstrap required, opened a new log
I20240503 08:11:36.321484 28549 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:36.321646 28549 ts_tablet_manager.cc:1367] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:11:36.321660 28551 ts_tablet_manager.cc:1367] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.016s
I20240503 08:11:36.322084 28546 ts_tablet_manager.cc:1398] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:11:36.322432 28551 raft_consensus.cc:348] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.322703 28551 raft_consensus.cc:374] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.322773 28551 raft_consensus.cc:729] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de33eb4315b4547889d1e6edae2f2dd, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.322774 28549 raft_consensus.cc:348] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.323041 28549 raft_consensus.cc:374] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.323119 28549 raft_consensus.cc:729] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.323027 28551 consensus_queue.cc:260] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.323339 28549 consensus_queue.cc:260] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.323716 28551 ts_tablet_manager.cc:1398] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:11:36.324427 28549 ts_tablet_manager.cc:1398] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:36.324703 28549 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed: Bootstrap starting.
I20240503 08:11:36.326227 28549 tablet_bootstrap.cc:654] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:36.327850 28549 tablet_bootstrap.cc:492] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed: No bootstrap required, opened a new log
I20240503 08:11:36.327970 28549 ts_tablet_manager.cc:1367] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:36.328658 28549 raft_consensus.cc:348] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.328887 28549 raft_consensus.cc:374] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.328977 28549 raft_consensus.cc:729] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d7c32cb9a241c99853a6639451c4ed, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.329175 28549 consensus_queue.cc:260] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.329612 28549 ts_tablet_manager.cc:1398] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:11:36.335523 28550 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b: No bootstrap required, opened a new log
I20240503 08:11:36.335719 28550 ts_tablet_manager.cc:1367] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.007s
I20240503 08:11:36.336732 28550 raft_consensus.cc:348] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.337026 28550 raft_consensus.cc:374] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:36.337122 28550 raft_consensus.cc:729] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.337340 28550 consensus_queue.cc:260] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.337733 28187 raft_consensus.cc:482] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:36.337919 28187 raft_consensus.cc:504] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.338752 28187 leader_election.cc:288] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:36.339118 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:36.339263 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:36.339361 27103 raft_consensus.cc:2454] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 0.
I20240503 08:11:36.339505 27495 raft_consensus.cc:2454] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 0.
I20240503 08:11:36.339591 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:36.339803 27624 raft_consensus.cc:2454] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 0.
I20240503 08:11:36.340026 27171 leader_election.cc:302] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80; no voters:
I20240503 08:11:36.340325 28187 raft_consensus.cc:2781] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:36.340318 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:36.340416 28187 raft_consensus.cc:482] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:36.340476 27753 raft_consensus.cc:2454] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 0.
I20240503 08:11:36.340514 28187 raft_consensus.cc:3037] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.340538 28550 ts_tablet_manager.cc:1398] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 08:11:36.342439 28187 raft_consensus.cc:504] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.343385 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:36.343348 28187 leader_election.cc:288] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 election: Requested vote from peers 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:36.343587 27753 raft_consensus.cc:3037] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.343638 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:36.343705 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:36.343848 27624 raft_consensus.cc:3037] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.343820 27103 raft_consensus.cc:3037] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.345683 27624 raft_consensus.cc:2454] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 1.
I20240503 08:11:36.346216 27753 raft_consensus.cc:2454] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 1.
I20240503 08:11:36.346374 27103 raft_consensus.cc:2454] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 1.
I20240503 08:11:36.346592 27171 leader_election.cc:302] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, 36fc7a106457453ba91f05b78a5d342b, d8d7c32cb9a241c99853a6639451c4ed; no voters:
I20240503 08:11:36.346604 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:36.346804 27495 raft_consensus.cc:3037] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.346855 28187 raft_consensus.cc:2781] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:36.347019 28187 raft_consensus.cc:686] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Becoming Leader. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Running, Role: LEADER
I20240503 08:11:36.347234 28187 consensus_queue.cc:237] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } }
I20240503 08:11:36.348766 27495 raft_consensus.cc:2454] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 1.
I20240503 08:11:36.350250 26959 catalog_manager.cc:5484] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: term changed from 0 to 1, leader changed from <none> to 36fc7a106457453ba91f05b78a5d342b (127.24.123.2). New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:36.459843 28437 raft_consensus.cc:482] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:36.460060 28437 raft_consensus.cc:504] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.461000 28437 leader_election.cc:288] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503)
I20240503 08:11:36.461426 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e17a88a118b4ce591a266fd62b3d7a7" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:36.461459 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e17a88a118b4ce591a266fd62b3d7a7" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:36.461661 27103 raft_consensus.cc:2454] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:36.461688 27753 raft_consensus.cc:2454] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:36.462203 27301 leader_election.cc:302] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: d8d7c32cb9a241c99853a6639451c4ed, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:36.462412 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e17a88a118b4ce591a266fd62b3d7a7" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:11:36.462512 28437 raft_consensus.cc:2781] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:36.462595 28437 raft_consensus.cc:482] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:36.462715 28437 raft_consensus.cc:3037] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.463264 27624 raft_consensus.cc:2454] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:36.465104 28437 raft_consensus.cc:504] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.466022 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e17a88a118b4ce591a266fd62b3d7a7" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:36.466140 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e17a88a118b4ce591a266fd62b3d7a7" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:36.466141 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e17a88a118b4ce591a266fd62b3d7a7" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:36.466226 27753 raft_consensus.cc:3037] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.466310 27103 raft_consensus.cc:3037] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.466310 27624 raft_consensus.cc:3037] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.468901 27753 raft_consensus.cc:2454] T 8e17a88a118b4ce591a266fd62b3d7a7 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:36.468984 27103 raft_consensus.cc:2454] T 8e17a88a118b4ce591a266fd62b3d7a7 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:36.468984 27624 raft_consensus.cc:2454] T 8e17a88a118b4ce591a266fd62b3d7a7 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:36.469560 27299 leader_election.cc:302] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2de33eb4315b4547889d1e6edae2f2dd, d8d7c32cb9a241c99853a6639451c4ed, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:36.470636 27234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e17a88a118b4ce591a266fd62b3d7a7" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:36.470782 28437 leader_election.cc:288] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 election: Requested vote from peers d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503)
I20240503 08:11:36.470763 28574 raft_consensus.cc:482] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:36.470889 27234 raft_consensus.cc:2454] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:36.470968 28437 raft_consensus.cc:2781] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:36.471105 28437 raft_consensus.cc:686] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Becoming Leader. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Running, Role: LEADER
I20240503 08:11:36.471242 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e17a88a118b4ce591a266fd62b3d7a7" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:36.471140 28574 raft_consensus.cc:504] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.471432 27235 raft_consensus.cc:3037] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.471413 28437 consensus_queue.cc:237] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } }
I20240503 08:11:36.472307 28574 leader_election.cc:288] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:36.472831 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb3a8dde7ec4d318b0961c8bad8032c" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" is_pre_election: true
I20240503 08:11:36.472824 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb3a8dde7ec4d318b0961c8bad8032c" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:11:36.473052 27103 raft_consensus.cc:2454] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:36.473855 27753 raft_consensus.cc:2454] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:36.473888 27235 raft_consensus.cc:2454] T 8e17a88a118b4ce591a266fd62b3d7a7 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:36.474725 27301 leader_election.cc:302] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: d8d7c32cb9a241c99853a6639451c4ed, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:36.474761 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb3a8dde7ec4d318b0961c8bad8032c" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b" is_pre_election: true
I20240503 08:11:36.474941 27235 raft_consensus.cc:2454] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:36.475054 28573 raft_consensus.cc:2781] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:36.475147 28573 raft_consensus.cc:482] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:36.475245 28573 raft_consensus.cc:3037] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.475275 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb3a8dde7ec4d318b0961c8bad8032c" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:11:36.475497 27495 raft_consensus.cc:2454] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 0.
I20240503 08:11:36.476308 26959 catalog_manager.cc:5484] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe8fdcdebbff4703a924f83646531115 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:36.478453 28573 raft_consensus.cc:504] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.479421 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb3a8dde7ec4d318b0961c8bad8032c" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:11:36.479597 27235 raft_consensus.cc:3037] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.479631 27753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb3a8dde7ec4d318b0961c8bad8032c" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:11:36.479835 27753 raft_consensus.cc:3037] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.479847 27495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb3a8dde7ec4d318b0961c8bad8032c" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:36.479918 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb3a8dde7ec4d318b0961c8bad8032c" candidate_uuid: "fe8fdcdebbff4703a924f83646531115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:36.480029 27495 raft_consensus.cc:3037] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.480149 27103 raft_consensus.cc:3037] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:36.482184 27235 raft_consensus.cc:2454] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:36.482498 27495 raft_consensus.cc:2454] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:36.482650 27103 raft_consensus.cc:2454] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:36.483153 27753 raft_consensus.cc:2454] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8fdcdebbff4703a924f83646531115 in term 1.
I20240503 08:11:36.483383 28573 leader_election.cc:288] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 election: Requested vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:36.483464 27301 leader_election.cc:302] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:36.483687 28573 raft_consensus.cc:2781] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:36.483846 28573 raft_consensus.cc:686] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Becoming Leader. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Running, Role: LEADER
I20240503 08:11:36.484090 28573 consensus_queue.cc:237] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } }
I20240503 08:11:36.487212 26959 catalog_manager.cc:5484] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe8fdcdebbff4703a924f83646531115 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:36.784487 28187 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:36.851321 28187 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:36.852361 28577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:36.852728 28577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:11:36.858161 28586 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:36.861268 28187 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:36.864104 28577 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:11:36.917878 28573 consensus_queue.cc:1035] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:36.959513 28573 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:36.966313 28573 consensus_queue.cc:1035] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:36.973289 28437 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:36.974460 28573 consensus_queue.cc:1035] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:36.977526 28573 consensus_queue.cc:1035] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:37.001940 28574 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:37.004262 28574 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:11:37.203159 27298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 72017.6us
INSERT report
rows total: 2000
time total: 344.669 ms
time per row: 0.172335 ms
I20240503 08:11:37.374251 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.7:41167
--local_ip_for_outbound_sockets=127.24.123.7
--tserver_master_addrs=127.24.123.62:40087
--webserver_port=45999
--webserver_interface=127.24.123.7
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:37.701004 28663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:37.701259 28663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:37.701364 28663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:37.708760 28663 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:37.708874 28663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:37.709026 28663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.7
I20240503 08:11:37.718384 28663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.7:41167
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.123.7
--webserver_port=45999
--tserver_master_addrs=127.24.123.62:40087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.7
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:37.719666 28663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:37.721246 28663 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:37.733445 28670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.733996 28671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.734522 28663 server_base.cc:1034] running on GCE node
W20240503 08:11:37.735138 28673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.735864 28663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:37.737023 28663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:37.738230 28663 hybrid_clock.cc:648] HybridClock initialized: now 1714723897738177 us; error 44 us; skew 500 ppm
I20240503 08:11:37.738505 28663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:37.741678 28663 webserver.cc:460] Webserver started at http://127.24.123.7:45999/ using document root <none> and password file <none>
I20240503 08:11:37.742453 28663 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:37.742576 28663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:37.747684 28663 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:11:37.750480 28679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.752077 28663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:11:37.752256 28663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1980d01e2e3e4840a8ddfca4d4e366b2"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-gkrw"
I20240503 08:11:37.753163 28663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.784363 28663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.785181 28663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:37.785506 28663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.787853 28663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:37.789652 28686 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 08:11:37.796056 28663 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:11:37.796150 28663 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20240503 08:11:37.796285 28663 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:11:37.799044 28663 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:11:37.799131 28663 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:37.799554 28686 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:37.821465 28686 log.cc:824] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:37.825661 28686 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:37.826426 28686 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:37.827548 28686 ts_tablet_manager.cc:1367] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.017s
I20240503 08:11:37.835717 28686 raft_consensus.cc:348] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:37.836812 28686 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: FOLLOWER
I20240503 08:11:37.837392 28686 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } }
I20240503 08:11:37.838716 28686 ts_tablet_manager.cc:1398] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20240503 08:11:37.857414 28663 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.7:41167
I20240503 08:11:37.857448 28795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.7:41167 every 8 connection(s)
I20240503 08:11:37.858681 28663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20240503 08:11:37.859805 28797 tablet.cc:2369] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:37.861723 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 28663
I20240503 08:11:37.862015 25068 test_util.cc:268] Using random seed: 1154606954
I20240503 08:11:37.871738 28796 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40087
I20240503 08:11:37.872009 28796 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:37.872756 28796 heartbeater.cc:511] Master 127.24.123.62:40087 requested a full tablet report, sending...
I20240503 08:11:37.875171 26958 ts_manager.cc:194] Re-registered known tserver with Master: 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167)
I20240503 08:11:37.876292 26958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.7:41297
I20240503 08:11:37.880343 25068 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 08:11:37.881018 25068 test_util.cc:268] Using random seed: 1154625945
I20240503 08:11:37.895467 25068 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 08:11:37.899562 25068 test_util.cc:268] Using random seed: 1154644486
I20240503 08:11:38.025802 28751 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:38.031076 28751 pending_rounds.cc:76] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Aborting all ops after (but not including) 0
I20240503 08:11:38.031225 28751 pending_rounds.cc:97] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240503 08:11:38.031355 28751 raft_consensus.cc:2866] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:11:38.032263 28796 heartbeater.cc:503] Master 127.24.123.62:40087 was elected leader, sending a full tablet report...
I20240503 08:11:38.094890 25068 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
W20240503 08:11:38.159691 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 54620.2us
I20240503 08:11:38.181363 25068 test_util.cc:268] Using random seed: 1154926269
I20240503 08:11:38.318571 25068 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 08:11:38.338150 25068 test_util.cc:268] Using random seed: 1155083064
W20240503 08:11:38.397114 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43185, user_credentials={real_user=slave}} blocked reactor thread for 65002us
I20240503 08:11:38.564420 25068 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240503 08:11:39.459594 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 114366us
W20240503 08:11:39.824764 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43185, user_credentials={real_user=slave}} blocked reactor thread for 74163.4us
W20240503 08:11:39.987298 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 77424us
W20240503 08:11:40.624433 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43185, user_credentials={real_user=slave}} blocked reactor thread for 43962.6us
W20240503 08:11:40.898941 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 94454.6us
W20240503 08:11:40.963245 27169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43185, user_credentials={real_user=slave}} blocked reactor thread for 108192us
W20240503 08:11:41.197819 27429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43185, user_credentials={real_user=slave}} blocked reactor thread for 61365.7us
W20240503 08:11:41.733225 27169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 93551.3us
W20240503 08:11:41.735839 27171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 99098.8us
I20240503 08:11:42.423807 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:42.468092 27213 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:42.512358 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:42.629434 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:42.639075 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:42.726990 27084 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:11:42.772017 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 52235.8us
W20240503 08:11:42.774528 27428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 50635.4us
I20240503 08:11:42.790643 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:11:44.768641 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 97013.3us
I20240503 08:11:45.016094 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:45.131217 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:45.311601 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:45.328471 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:45.363713 27084 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:11:45.537099 27213 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 08:11:45.537536 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 78962.5us
I20240503 08:11:45.610249 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:11:45.897399 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 95034.3us
W20240503 08:11:46.209029 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 53358.7us
W20240503 08:11:46.300248 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 44413.2us
W20240503 08:11:46.462929 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 71493.1us
W20240503 08:11:46.564126 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 46057.5us
I20240503 08:11:46.768306 27235 consensus_queue.cc:237] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:46.772490 27100 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 08:11:46.799811 27623 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 08:11:46.803406 27495 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 08:11:46.808606 29168 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:11:46.809314 29091 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:11:46.812011 27752 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 08:11:46.813129 29091 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:11:46.842633 29091 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:11:46.861846 29091 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:46.872900 26961 catalog_manager.cc:5484] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: config changed from index -1 to 84, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:46.885557 27753 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:46.889796 27101 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:46.892329 27495 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
W20240503 08:11:46.908964 27171 consensus_peers.cc:485] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c364e9ddad74e8e882df9080d9d5372. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:46.925935 27623 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.509291 27101 consensus_queue.cc:237] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:47.546991 27755 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
W20240503 08:11:47.546636 27040 consensus_peers.cc:485] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8c93cddbbf84976baa817ab0d98a811. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:47.547353 29209 ts_tablet_manager.cc:902] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503)
I20240503 08:11:47.547662 29163 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240503 08:11:47.549616 27621 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240503 08:11:47.550168 29163 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240503 08:11:47.565759 29045 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index -1 to 67, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.566695 29209 tablet_copy_client.cc:308] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:33503
I20240503 08:11:47.566804 27624 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index -1 to 67, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.574115 26955 catalog_manager.cc:5484] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index -1 to 67, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.641731 27753 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index -1 to 67, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.659782 27255 tablet_copy_service.cc:140] P 36fc7a106457453ba91f05b78a5d342b: Received BeginTabletCopySession request for tablet 4c364e9ddad74e8e882df9080d9d5372 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:34771)
I20240503 08:11:47.660073 27255 tablet_copy_service.cc:161] P 36fc7a106457453ba91f05b78a5d342b: Beginning new tablet copy session on tablet 4c364e9ddad74e8e882df9080d9d5372 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:34771: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-4c364e9ddad74e8e882df9080d9d5372
I20240503 08:11:47.665612 27255 tablet_copy_source_session.cc:205] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:47.669256 29209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c364e9ddad74e8e882df9080d9d5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:47.677747 29209 tablet_copy_client.cc:783] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:47.678182 29209 tablet_copy_client.cc:647] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:47.683320 29209 tablet_copy_client.cc:523] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:47.687866 29209 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:47.694350 27234 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240503 08:11:47.715821 27495 consensus_queue.cc:237] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:47.719986 27365 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240503 08:11:47.721212 29181 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 08:11:47.727787 27491 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240503 08:11:47.753955 27624 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240503 08:11:47.775151 29181 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 08:11:47.775243 29045 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240503 08:11:47.817469 27233 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index -1 to 67, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.837896 29181 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.847306 27360 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.851298 26955 catalog_manager.cc:5484] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 reported cstate change: config changed from index -1 to 51, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:47.866403 27753 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240503 08:11:47.870859 29045 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240503 08:11:47.877926 29181 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 08:11:47.889218 27753 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.903748 27623 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:47.952004 29241 ts_tablet_manager.cc:902] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:47.982445 27495 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index -1 to 67, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:48.009297 29241 tablet_copy_client.cc:308] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39261
I20240503 08:11:48.011358 27104 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
W20240503 08:11:48.026711 27431 consensus_peers.cc:485] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f09f0689f0714d8c8f45e24bd9adb533. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:48.072284 27104 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } }
I20240503 08:11:48.099558 29180 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 08:11:48.102378 27622 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:48.103127 29163 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:48.112007 28751 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:48.112628 29250 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:48.115670 27124 tablet_copy_service.cc:140] P f4d3aef6d5624cf4adc3859dbadccb80: Received BeginTabletCopySession request for tablet e8c93cddbbf84976baa817ab0d98a811 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:34495)
I20240503 08:11:48.115911 27124 tablet_copy_service.cc:161] P f4d3aef6d5624cf4adc3859dbadccb80: Beginning new tablet copy session on tablet e8c93cddbbf84976baa817ab0d98a811 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:34495: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-e8c93cddbbf84976baa817ab0d98a811
I20240503 08:11:48.120437 27124 tablet_copy_source_session.cc:205] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:48.123699 27365 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:48.129904 29241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c93cddbbf84976baa817ab0d98a811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:48.135571 27493 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:11:48.144327 29241 tablet_copy_client.cc:783] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:48.144951 29241 tablet_copy_client.cc:647] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:48.148097 29163 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:48.175408 29163 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:48.185994 29045 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
I20240503 08:11:48.191311 28751 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
I20240503 08:11:48.191807 27488 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
I20240503 08:11:48.196403 26959 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: config changed from index -1 to 2, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New cstate: current_term: 2 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:48.198410 27099 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
W20240503 08:11:48.215008 27037 consensus_peers.cc:485] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Couldn't send request to peer 36fc7a106457453ba91f05b78a5d342b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8d5c3e2f8e6428aa457ba21d536cea5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:48.224243 29241 tablet_copy_client.cc:523] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:48.227648 29241 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:48.280437 29271 ts_tablet_manager.cc:902] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:11:48.322708 27365 consensus_queue.cc:237] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:48.325054 29271 tablet_copy_client.cc:308] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:41395
I20240503 08:11:48.327246 27625 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 08:11:48.328003 27495 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 08:11:48.328003 29211 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 08:11:48.328899 28949 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 08:11:48.334168 28949 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:48.335067 27495 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:48.339711 27625 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:48.340538 27360 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
I20240503 08:11:48.341034 26955 catalog_manager.cc:5484] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index -1 to 69, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:48.410195 27515 tablet_copy_service.cc:140] P 19fb518a6ede476f809c743054941e97: Received BeginTabletCopySession request for tablet f09f0689f0714d8c8f45e24bd9adb533 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:60417)
I20240503 08:11:48.410441 27515 tablet_copy_service.cc:161] P 19fb518a6ede476f809c743054941e97: Beginning new tablet copy session on tablet f09f0689f0714d8c8f45e24bd9adb533 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:60417: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-f09f0689f0714d8c8f45e24bd9adb533
I20240503 08:11:48.414040 27515 tablet_copy_source_session.cc:205] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:48.428320 29271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09f0689f0714d8c8f45e24bd9adb533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:48.435335 29271 tablet_copy_client.cc:783] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:48.435588 29271 tablet_copy_client.cc:647] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:48.443924 29241 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=68 overwritten=0 applied=68 ignored=0} inserts{seen=740 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:48.444404 29241 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:48.445034 29241 ts_tablet_manager.cc:1367] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 0.217s user 0.109s sys 0.010s
I20240503 08:11:48.446254 29241 raft_consensus.cc:348] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:48.446751 29241 raft_consensus.cc:729] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:11:48.447088 29241 consensus_queue.cc:260] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 68, Last appended: 1.68, Last appended by leader: 68, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:48.451591 29241 ts_tablet_manager.cc:1398] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:11:48.452688 27124 tablet_copy_service.cc:342] P f4d3aef6d5624cf4adc3859dbadccb80: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-e8c93cddbbf84976baa817ab0d98a811 received from {username='slave'} at 127.24.123.7:34495
I20240503 08:11:48.452862 27124 tablet_copy_service.cc:433] P f4d3aef6d5624cf4adc3859dbadccb80: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-e8c93cddbbf84976baa817ab0d98a811 on tablet e8c93cddbbf84976baa817ab0d98a811 with peer 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:48.473557 27100 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 08:11:48.476068 27755 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 08:11:48.479050 27625 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
I20240503 08:11:48.503547 29271 tablet_copy_client.cc:523] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:48.507720 29271 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:48.508046 28751 raft_consensus.cc:1203] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.66->[1.67-1.69] Dedup: 1.68->[1.69-1.69]
I20240503 08:11:48.581552 29211 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 08:11:48.581852 29011 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
W20240503 08:11:48.613008 27301 consensus_peers.cc:485] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 921b4706d2bc4693ae5a0a10b310d5d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:48.633669 29291 ts_tablet_manager.cc:902] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: Initiating tablet copy from peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:48.645558 27101 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:48.655819 29209 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=101 overwritten=0 applied=101 ignored=0} inserts{seen=773 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:48.657408 29209 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:48.657768 29209 ts_tablet_manager.cc:1367] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 0.970s user 0.178s sys 0.022s
I20240503 08:11:48.658543 29209 raft_consensus.cc:348] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:48.666440 29209 raft_consensus.cc:729] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:11:48.666709 29209 consensus_queue.cc:260] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 101, Last appended: 1.101, Last appended by leader: 101, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:48.675598 29209 ts_tablet_manager.cc:1398] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240503 08:11:48.676633 27255 tablet_copy_service.cc:342] P 36fc7a106457453ba91f05b78a5d342b: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-4c364e9ddad74e8e882df9080d9d5372 received from {username='slave'} at 127.24.123.7:34771
I20240503 08:11:48.676828 27255 tablet_copy_service.cc:433] P 36fc7a106457453ba91f05b78a5d342b: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-4c364e9ddad74e8e882df9080d9d5372 on tablet 4c364e9ddad74e8e882df9080d9d5372 with peer 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:48.684437 27755 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:48.702941 29268 raft_consensus.cc:1050] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:48.704767 29268 consensus_queue.cc:237] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
W20240503 08:11:48.710690 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 74782.2us
I20240503 08:11:48.755364 29209 ts_tablet_manager.cc:902] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:11:48.776113 27233 consensus_queue.cc:237] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.113, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: true } }
I20240503 08:11:48.778925 27625 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 08:11:48.781244 29250 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
W20240503 08:11:48.782172 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 69341.1us
I20240503 08:11:48.782263 28751 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 08:11:48.782752 27753 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 08:11:48.782807 27232 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 08:11:48.783012 29250 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240503 08:11:48.783298 29163 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240503 08:11:48.783829 29163 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240503 08:11:48.797364 29250 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 67 to 73, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:48.798007 27232 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 67 to 73, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:48.799295 27360 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240503 08:11:48.799788 29167 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240503 08:11:48.801584 28751 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 67 to 73, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:48.805294 28750 raft_consensus.cc:1203] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.83->[1.84-1.118] Dedup: 1.101->[1.102-1.118]
I20240503 08:11:48.813318 26959 catalog_manager.cc:5484] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: config changed from index 67 to 73, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:48.818259 27621 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 67 to 73, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:48.821689 29291 tablet_copy_client.cc:308] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39261
I20240503 08:11:48.822461 27124 tablet_copy_service.cc:140] P f4d3aef6d5624cf4adc3859dbadccb80: Received BeginTabletCopySession request for tablet b8d5c3e2f8e6428aa457ba21d536cea5 from peer 36fc7a106457453ba91f05b78a5d342b ({username='slave'} at 127.24.123.2:58467)
I20240503 08:11:48.822646 27124 tablet_copy_service.cc:161] P f4d3aef6d5624cf4adc3859dbadccb80: Beginning new tablet copy session on tablet b8d5c3e2f8e6428aa457ba21d536cea5 from peer 36fc7a106457453ba91f05b78a5d342b at {username='slave'} at 127.24.123.2:58467: session id = 36fc7a106457453ba91f05b78a5d342b-b8d5c3e2f8e6428aa457ba21d536cea5
I20240503 08:11:48.825353 27124 tablet_copy_source_session.cc:205] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:48.831528 29291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d5c3e2f8e6428aa457ba21d536cea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:48.834434 27624 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240503 08:11:48.835244 27490 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240503 08:11:48.839290 29167 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240503 08:11:48.843847 27490 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 08:11:48.844774 29167 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240503 08:11:48.846117 27753 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
W20240503 08:11:48.847144 27171 consensus_peers.cc:485] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b -> Peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Couldn't send request to peer f4d3aef6d5624cf4adc3859dbadccb80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e4a8224c7c1430a9354e58bd16c7abd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:48.849824 29019 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240503 08:11:48.850667 29091 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: true } } }
I20240503 08:11:48.857437 27751 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: true } } }
I20240503 08:11:48.862980 27624 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: true } } }
I20240503 08:11:48.868211 29250 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240503 08:11:48.869277 26955 catalog_manager.cc:5484] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed reported cstate change: config changed from index -1 to 114, NON_VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) added. New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: true } } }
I20240503 08:11:48.881752 27490 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 67 to 73, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:48.906956 29209 tablet_copy_client.cc:308] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:43977
I20240503 08:11:48.909314 27752 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 67 to 73, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:48.922839 27360 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: true } } }
I20240503 08:11:48.933414 27490 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index -1 to 114, NON_VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) added. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: true } } }
I20240503 08:11:48.958616 27099 consensus_queue.cc:237] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:48.962430 28750 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 08:11:48.963060 29250 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 08:11:48.980228 27233 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 08:11:48.977440 27624 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 08:11:48.985605 29310 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 08:11:48.992821 29045 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index 73 to 76, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:48.996583 27385 tablet_copy_service.cc:140] P fe8fdcdebbff4703a924f83646531115: Received BeginTabletCopySession request for tablet 921b4706d2bc4693ae5a0a10b310d5d4 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:40565)
I20240503 08:11:48.996827 27385 tablet_copy_service.cc:161] P fe8fdcdebbff4703a924f83646531115: Beginning new tablet copy session on tablet 921b4706d2bc4693ae5a0a10b310d5d4 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:40565: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-921b4706d2bc4693ae5a0a10b310d5d4
I20240503 08:11:48.999620 26950 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e8c93cddbbf84976baa817ab0d98a811 with cas_config_opid_index 73: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:49.000450 27385 tablet_copy_source_session.cc:205] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:49.001116 28750 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 73 to 76, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.002306 29209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921b4706d2bc4693ae5a0a10b310d5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:49.004753 26960 catalog_manager.cc:5484] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: config changed from index 73 to 76, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:49.008628 29209 tablet_copy_client.cc:783] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:49.008854 29209 tablet_copy_client.cc:647] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:49.010248 27733 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c93cddbbf84976baa817ab0d98a811 with delete_type TABLET_DATA_TOMBSTONED (TS d8d7c32cb9a241c99853a6639451c4ed not found in new config with opid_index 76) from {username='slave'} at 127.0.0.1:42730
I20240503 08:11:49.011544 27489 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 08:11:49.018077 27624 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 73 to 76, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.025300 29209 tablet_copy_client.cc:523] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:49.029112 29209 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:49.044431 29268 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 08:11:49.056295 29291 tablet_copy_client.cc:783] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:49.091631 27625 consensus_queue.cc:237] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
W20240503 08:11:49.108106 27298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 45700us
I20240503 08:11:49.110296 27365 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 08:11:49.112370 29293 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 08:11:49.126137 29045 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 08:11:49.127357 27492 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 08:11:49.150317 28865 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 08:11:49.163522 27751 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 08:11:49.170164 29331 tablet_replica.cc:330] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed: stopping tablet replica
I20240503 08:11:49.058162 27233 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 73 to 76, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.094462 27235 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 08:11:49.101914 29091 raft_consensus.cc:1050] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:49.106818 29019 raft_consensus.cc:1050] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:49.148437 29167 raft_consensus.cc:1050] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:49.174154 28865 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 08:11:49.177070 29293 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 08:11:49.180261 29331 raft_consensus.cc:2229] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:49.187589 27492 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 73 to 76, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.204425 29300 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:49.216431 26955 catalog_manager.cc:5484] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index -1 to 99, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:49.225137 29331 raft_consensus.cc:2258] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:49.228473 27365 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:49.258024 27232 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:49.274989 29331 ts_tablet_manager.cc:1875] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:49.287950 29291 tablet_copy_client.cc:647] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:49.294194 29091 consensus_queue.cc:237] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.121, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
W20240503 08:11:49.296896 29167 raft_consensus.cc:1054] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: Unable to promote non-voter 1980d01e2e3e4840a8ddfca4d4e366b2: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }.
Pending config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
W20240503 08:11:49.297523 29019 raft_consensus.cc:1054] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: Unable to promote non-voter 1980d01e2e3e4840a8ddfca4d4e366b2: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }.
Pending config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:49.302449 29331 ts_tablet_manager.cc:1888] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.74
I20240503 08:11:49.302620 29331 log.cc:1196] T e8c93cddbbf84976baa817ab0d98a811 P d8d7c32cb9a241c99853a6639451c4ed: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/e8c93cddbbf84976baa817ab0d98a811
I20240503 08:11:49.303746 26950 catalog_manager.cc:4830] TS d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465): tablet e8c93cddbbf84976baa817ab0d98a811 (table rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]) successfully deleted
W20240503 08:11:49.311614 27171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 154673us
W20240503 08:11:49.328212 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 70932.2us
I20240503 08:11:49.352355 27494 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 08:11:49.353607 27489 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:49.364109 28750 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 08:11:49.364646 29347 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240503 08:11:49.366221 27751 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 08:11:49.366719 29347 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240503 08:11:49.369895 29337 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 119, Time since last communication: 0.000s
I20240503 08:11:49.384269 29347 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Committing config change with OpId 1.122: config changed from index 84 to 122, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.396608 27102 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 08:11:49.404868 29334 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240503 08:11:49.407019 28750 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 84 to 122, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.408118 29291 tablet_copy_client.cc:523] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:49.407809 27751 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 84 to 122, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.412984 26955 catalog_manager.cc:5484] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 reported cstate change: config changed from index 84 to 122, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.417222 29291 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
I20240503 08:11:49.422933 27621 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 08:11:49.425987 27101 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 84 to 122, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.442754 29352 ts_tablet_manager.cc:902] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: Initiating tablet copy from peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503)
W20240503 08:11:49.459908 27561 consensus_peers.cc:485] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08b4767b5b4f4fb0bfc73c35a408a5ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:49.460606 27753 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:49.488976 29334 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240503 08:11:49.494324 27621 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 84 to 122, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.505183 27493 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 84 to 122, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.573570 29291 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:49.574383 27234 consensus_queue.cc:237] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.124, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:49.577298 27750 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240503 08:11:49.577854 29347 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s
I20240503 08:11:49.580982 27625 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240503 08:11:49.584070 29291 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap complete.
I20240503 08:11:49.588603 28751 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240503 08:11:49.586097 29347 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s
I20240503 08:11:49.625416 29091 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s
I20240503 08:11:49.626339 29291 ts_tablet_manager.cc:1367] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.209s user 0.007s sys 0.003s
I20240503 08:11:49.629951 27490 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240503 08:11:49.630069 29091 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Committing config change with OpId 1.125: config changed from index 122 to 125, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.627208 29291 raft_consensus.cc:348] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } }
I20240503 08:11:49.656349 29291 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: LEARNER
I20240503 08:11:49.656826 28750 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 122 to 125, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.657094 29291 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } }
I20240503 08:11:49.658360 29291 ts_tablet_manager.cc:1398] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240503 08:11:49.664455 27752 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 122 to 125, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.670898 26947 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c364e9ddad74e8e882df9080d9d5372 with cas_config_opid_index 122: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:49.675220 26955 catalog_manager.cc:5484] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed reported cstate change: config changed from index 122 to 125, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.680519 27084 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c364e9ddad74e8e882df9080d9d5372 with delete_type TABLET_DATA_TOMBSTONED (TS f4d3aef6d5624cf4adc3859dbadccb80 not found in new config with opid_index 125) from {username='slave'} at 127.0.0.1:57128
I20240503 08:11:49.695600 29019 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 122, Time since last communication: 0.008s
I20240503 08:11:49.703542 27124 tablet_copy_service.cc:342] P f4d3aef6d5624cf4adc3859dbadccb80: Request end of tablet copy session 36fc7a106457453ba91f05b78a5d342b-b8d5c3e2f8e6428aa457ba21d536cea5 received from {username='slave'} at 127.24.123.2:58467
I20240503 08:11:49.703713 27124 tablet_copy_service.cc:433] P f4d3aef6d5624cf4adc3859dbadccb80: ending tablet copy session 36fc7a106457453ba91f05b78a5d342b-b8d5c3e2f8e6428aa457ba21d536cea5 on tablet b8d5c3e2f8e6428aa457ba21d536cea5 with peer 36fc7a106457453ba91f05b78a5d342b
W20240503 08:11:49.710549 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 78199.4us
I20240503 08:11:49.724925 29371 tablet_replica.cc:330] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: stopping tablet replica
I20240503 08:11:49.734151 27490 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 122 to 125, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.740427 29209 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=73 overwritten=0 applied=73 ignored=0} inserts{seen=758 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:49.740890 29209 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:49.741155 29209 ts_tablet_manager.cc:1367] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 0.712s user 0.101s sys 0.027s
I20240503 08:11:49.741982 29209 raft_consensus.cc:348] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:49.742250 29209 raft_consensus.cc:729] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:11:49.745332 29371 raft_consensus.cc:2229] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:49.751022 29371 pending_rounds.cc:62] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: Trying to abort 1 pending ops.
I20240503 08:11:49.751354 29371 pending_rounds.cc:68] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: Aborting op as it isn't in flight: id { term: 1 index: 123 } timestamp: 7023509133523525632 op_type: WRITE_OP write_request { tablet_id: "4c364e9ddad74e8e882df9080d9d5372" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\r\270\034\n\352\271\3562\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""0c958f8d5e06f0de" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509133273948160 authz_token { token_data: "\010\346\272\322\261\006\"3\n\005slave\022*\n 53ec6f491e9d4c4ba45ce9e65edfcd3c\020\001\030\001 \001(\001" signature: "<redacted>""Z\2108&U(r\255\031i\036\332\376$\016J\002\005h\362\254\247T7\255\241\t\2016\003\224\235\303Y\3357\376[g\317\0014Q\212\2577\rY\t\003gn9r\'\344o\224{\320\351H\221\311" signing_key_seq_num: 0 } } request_id { client_id: "1bb282e8295f40939a052b834fcdf539" seq_no: 374 first_incomplete_seq_no: 374 attempt_no: 0 }
I20240503 08:11:49.753082 29371 raft_consensus.cc:2258] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:49.753641 29366 ts_tablet_manager.cc:902] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:49.759111 29371 ts_tablet_manager.cc:1875] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:49.774731 29371 ts_tablet_manager.cc:1888] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.123
I20240503 08:11:49.775070 29371 log.cc:1196] T 4c364e9ddad74e8e882df9080d9d5372 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c364e9ddad74e8e882df9080d9d5372
I20240503 08:11:49.780360 26950 catalog_manager.cc:4830] TS f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): tablet 4c364e9ddad74e8e882df9080d9d5372 (table rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]) successfully deleted
I20240503 08:11:49.806329 29366 tablet_copy_client.cc:308] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43185
I20240503 08:11:49.742486 29209 consensus_queue.cc:260] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 73, Last appended: 1.73, Last appended by leader: 73, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:49.829352 29209 ts_tablet_manager.cc:1398] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.088s user 0.004s sys 0.000s
I20240503 08:11:49.839166 27385 tablet_copy_service.cc:342] P fe8fdcdebbff4703a924f83646531115: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-921b4706d2bc4693ae5a0a10b310d5d4 received from {username='slave'} at 127.24.123.7:40565
I20240503 08:11:49.839365 27385 tablet_copy_service.cc:433] P fe8fdcdebbff4703a924f83646531115: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-921b4706d2bc4693ae5a0a10b310d5d4 on tablet 921b4706d2bc4693ae5a0a10b310d5d4 with peer 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:49.857010 29271 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=684 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:49.872375 27234 raft_consensus.cc:1203] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 08:11:49.884877 29271 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:49.885710 29271 ts_tablet_manager.cc:1367] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 1.378s user 0.104s sys 0.000s
I20240503 08:11:49.891155 29271 raft_consensus.cc:348] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:49.892392 29271 raft_consensus.cc:729] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:11:49.892709 29271 consensus_queue.cc:260] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 1.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:49.893266 27645 tablet_copy_service.cc:140] P 2de33eb4315b4547889d1e6edae2f2dd: Received BeginTabletCopySession request for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:42183)
I20240503 08:11:49.893486 27645 tablet_copy_service.cc:161] P 2de33eb4315b4547889d1e6edae2f2dd: Beginning new tablet copy session on tablet 08b4767b5b4f4fb0bfc73c35a408a5ac from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:42183: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-08b4767b5b4f4fb0bfc73c35a408a5ac
I20240503 08:11:49.894568 29271 ts_tablet_manager.cc:1398] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:11:49.897063 27645 tablet_copy_source_session.cc:205] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:49.900363 27515 tablet_copy_service.cc:342] P 19fb518a6ede476f809c743054941e97: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-f09f0689f0714d8c8f45e24bd9adb533 received from {username='slave'} at 127.24.123.7:60417
I20240503 08:11:49.900527 27515 tablet_copy_service.cc:433] P 19fb518a6ede476f809c743054941e97: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-f09f0689f0714d8c8f45e24bd9adb533 on tablet f09f0689f0714d8c8f45e24bd9adb533 with peer 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:49.904057 29366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:49.916476 29366 tablet_copy_client.cc:783] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:49.916725 29366 tablet_copy_client.cc:647] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:49.934566 29366 tablet_copy_client.cc:523] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:49.937988 29366 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:49.948997 29352 tablet_copy_client.cc:308] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:33503
W20240503 08:11:49.949944 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 45410.2us
I20240503 08:11:49.957679 27255 tablet_copy_service.cc:140] P 36fc7a106457453ba91f05b78a5d342b: Received BeginTabletCopySession request for tablet 8e4a8224c7c1430a9354e58bd16c7abd from peer f4d3aef6d5624cf4adc3859dbadccb80 ({username='slave'} at 127.24.123.1:39693)
I20240503 08:11:49.957859 27255 tablet_copy_service.cc:161] P 36fc7a106457453ba91f05b78a5d342b: Beginning new tablet copy session on tablet 8e4a8224c7c1430a9354e58bd16c7abd from peer f4d3aef6d5624cf4adc3859dbadccb80 at {username='slave'} at 127.24.123.1:39693: session id = f4d3aef6d5624cf4adc3859dbadccb80-8e4a8224c7c1430a9354e58bd16c7abd
I20240503 08:11:49.957813 27621 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 122 to 125, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:49.960767 27255 tablet_copy_source_session.cc:205] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 08:11:50.092808 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 119128us
I20240503 08:11:50.138689 29352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4a8224c7c1430a9354e58bd16c7abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:50.171937 28749 raft_consensus.cc:1203] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.50->[1.51-1.57] Dedup: 1.53->[1.54-1.57]
W20240503 08:11:50.258528 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 137885us
I20240503 08:11:50.288280 29366 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=104 overwritten=0 applied=104 ignored=0} inserts{seen=735 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:50.288853 29366 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:50.289139 29366 ts_tablet_manager.cc:1367] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 0.351s user 0.145s sys 0.009s
I20240503 08:11:50.289841 29366 raft_consensus.cc:348] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:50.290114 29366 raft_consensus.cc:729] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:11:50.290323 29366 consensus_queue.cc:260] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 104, Last appended: 1.104, Last appended by leader: 104, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:50.290912 29366 ts_tablet_manager.cc:1398] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:11:50.304160 28981 raft_consensus.cc:1050] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:50.305959 28981 consensus_queue.cc:237] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:50.309674 28749 raft_consensus.cc:1203] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.68->[1.69-1.80] Dedup: 1.73->[1.74-1.80]
I20240503 08:11:50.313040 28750 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 08:11:50.313608 27750 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 08:11:50.313943 29181 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 08:11:50.316128 29181 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 08:11:50.320659 27361 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 08:11:50.321393 29181 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 08:11:50.326748 29181 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index 51 to 59, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.329186 27361 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 51 to 59, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.338603 27645 tablet_copy_service.cc:342] P 2de33eb4315b4547889d1e6edae2f2dd: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-08b4767b5b4f4fb0bfc73c35a408a5ac received from {username='slave'} at 127.24.123.7:42183
I20240503 08:11:50.338786 27645 tablet_copy_service.cc:433] P 2de33eb4315b4547889d1e6edae2f2dd: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-08b4767b5b4f4fb0bfc73c35a408a5ac on tablet 08b4767b5b4f4fb0bfc73c35a408a5ac with peer 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:50.338335 27750 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 51 to 59, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.339610 28749 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 51 to 59, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.341054 27625 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 08:11:50.345121 26955 catalog_manager.cc:5484] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 51 to 59, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.349514 28981 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 08:11:50.354460 27621 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 51 to 59, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.368068 29386 raft_consensus.cc:1050] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:50.369004 29386 consensus_queue.cc:237] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:50.373920 28749 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 08:11:50.375180 29201 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 08:11:50.378296 27755 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 08:11:50.378381 27489 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 08:11:50.379251 29383 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 08:11:50.388686 28949 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 08:11:50.404356 29383 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 69 to 82, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.408871 27755 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 69 to 82, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.408869 28749 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 69 to 82, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.419309 27493 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 69 to 82, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.424412 29352 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.24.123.2:58467 (request call id 240) took 1070 ms. Trace:
I20240503 08:11:50.428948 27621 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 08:11:50.431296 27098 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 08:11:50.431270 29383 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 08:11:50.435463 28981 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 08:11:50.424860 29352 rpcz_store.cc:276] 0503 08:11:49.353450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:11:49.353606 (+ 156us) service_pool.cc:225] Handling call
0503 08:11:49.442841 (+ 89235us) ts_tablet_manager.cc:903] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: Initiating tablet copy from peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503)
0503 08:11:50.134703 (+691862us) tablet_copy_client.cc:353] Tablet copy session begun
0503 08:11:50.142390 (+ 7687us) tablet_metadata.cc:641] Metadata flushed
0503 08:11:50.142520 (+ 130us) tablet_copy_client.cc:482] Wrote new tablet metadata
0503 08:11:50.424067 (+281547us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 08:11:50.424245 (+ 178us) tablet_replica.cc:205] Creating consensus instance
0503 08:11:50.424339 (+ 94us) ts_tablet_manager.cc:958] Replica registered.
0503 08:11:50.424366 (+ 27us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":106,"tablet-copy.queue_time_us":41064,"thread_start_us":523971,"threads_started":5}
I20240503 08:11:50.438216 27098 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 08:11:50.448633 28949 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 08:11:50.449076 27101 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 51 to 59, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.452580 29352 tablet_copy_client.cc:783] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:50.455636 29352 tablet_copy_client.cc:647] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:50.495414 27624 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 69 to 82, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.522722 29352 tablet_copy_client.cc:523] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:50.521981 27104 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 69 to 82, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.533051 29352 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap starting.
W20240503 08:11:50.556380 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 119305us
I20240503 08:11:50.608153 26958 catalog_manager.cc:5484] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed reported cstate change: config changed from index 69 to 82, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.621366 27493 consensus_queue.cc:237] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:50.621304 27361 consensus_queue.cc:237] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
W20240503 08:11:50.623961 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 74575.3us
I20240503 08:11:50.624197 27622 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 08:11:50.624629 28749 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 08:11:50.625914 27490 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 08:11:50.626595 29228 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 08:11:50.626979 27755 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 08:11:50.626982 28981 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 08:11:50.627801 28749 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 08:11:50.629263 29315 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:11:50.629701 29407 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 82, Time since last communication: 0.000s
I20240503 08:11:50.635659 28949 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.636004 29351 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index 59 to 64, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.636526 27622 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 59 to 64, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.640209 27753 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.643041 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f09f0689f0714d8c8f45e24bd9adb533 with cas_config_opid_index 59: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:50.646240 26955 catalog_manager.cc:5484] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 reported cstate change: config changed from index 59 to 64, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:50.649824 26957 catalog_manager.cc:5484] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed reported cstate change: config changed from index 82 to 84, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.652853 28749 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.653512 27755 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 08:11:50.655006 27605 tablet_service.cc:1514] Processing DeleteTablet for tablet 921b4706d2bc4693ae5a0a10b310d5d4 with delete_type TABLET_DATA_TOMBSTONED (TS 2de33eb4315b4547889d1e6edae2f2dd not found in new config with opid_index 84) from {username='slave'} at 127.0.0.1:33064
I20240503 08:11:50.656077 28750 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 59 to 64, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.659219 27102 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 08:11:50.668629 26947 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 921b4706d2bc4693ae5a0a10b310d5d4 with cas_config_opid_index 82: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:50.670794 27344 tablet_service.cc:1514] Processing DeleteTablet for tablet f09f0689f0714d8c8f45e24bd9adb533 with delete_type TABLET_DATA_TOMBSTONED (TS fe8fdcdebbff4703a924f83646531115 not found in new config with opid_index 64) from {username='slave'} at 127.0.0.1:60962
I20240503 08:11:50.672375 29386 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:11:50.677284 28949 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 08:11:50.682025 29250 raft_consensus.cc:1050] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: attempting to promote NON_VOTER 36fc7a106457453ba91f05b78a5d342b to VOTER
I20240503 08:11:50.684484 27491 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.685405 27100 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 08:11:50.686900 29250 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } }
I20240503 08:11:50.701504 29419 tablet_replica.cc:330] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: stopping tablet replica
I20240503 08:11:50.702164 29419 raft_consensus.cc:2229] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:50.702569 29419 pending_rounds.cc:62] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: Trying to abort 1 pending ops.
I20240503 08:11:50.702731 29419 pending_rounds.cc:68] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: Aborting op as it isn't in flight: id { term: 1 index: 63 } timestamp: 7023509137883410432 op_type: WRITE_OP write_request { tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\243Z\036S\315)M\037\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5c4c6a5139aff4b5" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509137595236352 authz_token { token_data: "\010\346\272\322\261\006\"3\n\005slave\022*\n e89fa94dfc184007962228c3c499e7e5\020\001\030\001 \001(\001" signature: "<redacted>""7\377\304p1\247\021\245\332\203<\264\254\300hM\233\017\004\303\302?\035\271pJ\303\231w\'\372v\177\373I\031\222T\350p eH\030\217\221\2416\375\262\231\010\273\277\303^\236PyZ\025\225\373Y" signing_key_seq_num: 0 } } request_id { client_id: "dc47dab7f58940c2ac7feeaed7e55991" seq_no: 220 first_incomplete_seq_no: 220 attempt_no: 0 }
I20240503 08:11:50.703588 29419 raft_consensus.cc:2258] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:50.706323 29419 ts_tablet_manager.cc:1875] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:50.716895 28750 raft_consensus.cc:1203] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.98->[1.99-1.108] Dedup: 1.104->[1.105-1.108]
I20240503 08:11:50.720456 27098 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.729267 29419 ts_tablet_manager.cc:1888] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.63
I20240503 08:11:50.729502 29419 log.cc:1196] T f09f0689f0714d8c8f45e24bd9adb533 P fe8fdcdebbff4703a924f83646531115: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/f09f0689f0714d8c8f45e24bd9adb533
I20240503 08:11:50.739419 28751 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:11:50.739995 29310 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:50.744310 26947 catalog_manager.cc:4830] TS fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): tablet f09f0689f0714d8c8f45e24bd9adb533 (table rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]) successfully deleted
I20240503 08:11:50.745944 27362 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:11:50.746681 29310 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:50.752648 29418 tablet_replica.cc:330] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd: stopping tablet replica
I20240503 08:11:50.753235 29418 raft_consensus.cc:2229] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:50.753748 29418 raft_consensus.cc:2258] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:50.767270 27234 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 2 LEARNER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:11:50.773397 29418 ts_tablet_manager.cc:1875] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:50.805965 29418 ts_tablet_manager.cc:1888] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.83
I20240503 08:11:50.806138 29418 log.cc:1196] T 921b4706d2bc4693ae5a0a10b310d5d4 P 2de33eb4315b4547889d1e6edae2f2dd: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/wals/921b4706d2bc4693ae5a0a10b310d5d4
I20240503 08:11:50.808741 26948 catalog_manager.cc:4830] TS 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185): tablet 921b4706d2bc4693ae5a0a10b310d5d4 (table rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]) successfully deleted
I20240503 08:11:50.814076 29421 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:50.824378 29409 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 61, Time since last communication: 0.000s
I20240503 08:11:50.824949 29409 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 08:11:50.832873 27098 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 59 to 64, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.856475 27755 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 59 to 64, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
W20240503 08:11:50.871073 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 192045us
I20240503 08:11:50.880151 27621 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:11:50.919890 29360 raft_consensus.cc:1050] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:50.920981 29360 consensus_queue.cc:237] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:50.924826 27362 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240503 08:11:50.924883 27753 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240503 08:11:50.924981 28749 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240503 08:11:50.925383 29361 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 108, Time since last communication: 0.000s
I20240503 08:11:50.925845 29361 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 108, Time since last communication: 0.000s
I20240503 08:11:50.927177 29361 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 108, Time since last communication: 0.000s
I20240503 08:11:50.929915 29361 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index 99 to 110, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.932364 27361 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 99 to 110, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.936983 27753 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 99 to 110, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.936584 28749 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 99 to 110, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:50.940567 26961 catalog_manager.cc:5484] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index 99 to 110, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:50.944464 27235 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 108. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240503 08:11:50.946238 29360 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 108, Time since last communication: 0.000s
I20240503 08:11:50.951831 29420 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:50.952970 27235 raft_consensus.cc:1203] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.108->[1.109-1.110] Dedup: 1.109->[1.110-1.110]
I20240503 08:11:50.958706 27489 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:11:50.964548 27495 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 108. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240503 08:11:50.983172 27235 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 99 to 110, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:51.002457 29420 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:51.003994 29310 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:51.004670 27622 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:51.012696 27361 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:51.012996 29360 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 108, Time since last communication: 0.000s
I20240503 08:11:51.035604 27490 raft_consensus.cc:1203] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.108->[1.109-1.110] Dedup: 1.109->[1.110-1.110]
I20240503 08:11:51.036368 27490 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 99 to 110, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:51.076916 28750 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:51.114244 26957 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 2 to 3, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:51.159832 27233 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:51.237286 27625 consensus_queue.cc:237] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 1.115, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:51.241811 27362 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240503 08:11:51.244639 29360 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 114, Time since last communication: 0.000s
I20240503 08:11:51.244129 27750 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240503 08:11:51.247304 28750 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240503 08:11:51.247494 29360 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 114, Time since last communication: 0.000s
I20240503 08:11:51.242489 27493 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:51.255142 29360 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 114, Time since last communication: 0.000s
I20240503 08:11:51.267388 29391 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.116: config changed from index 110 to 116, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:51.272188 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac with cas_config_opid_index 110: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:51.274618 27362 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 110 to 116, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:51.279549 27755 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 110 to 116, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:51.280264 26957 catalog_manager.cc:5484] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 110 to 116, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:51.281001 28748 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 110 to 116, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:51.285862 27213 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac with delete_type TABLET_DATA_TOMBSTONED (TS 36fc7a106457453ba91f05b78a5d342b not found in new config with opid_index 116) from {username='slave'} at 127.0.0.1:50738
I20240503 08:11:51.319200 29454 tablet_replica.cc:330] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: stopping tablet replica
I20240503 08:11:51.319635 29454 raft_consensus.cc:2229] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:51.327136 29454 pending_rounds.cc:62] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: Trying to abort 1 pending ops.
I20240503 08:11:51.327459 29454 pending_rounds.cc:68] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: Aborting op as it isn't in flight: id { term: 1 index: 115 } timestamp: 7023509140348788736 op_type: WRITE_OP write_request { tablet_id: "08b4767b5b4f4fb0bfc73c35a408a5ac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000N\372\332\0007\035,\010\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c69993f9198387a3" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509140034715648 authz_token { token_data: "\010\345\272\322\261\006\"3\n\005slave\022*\n 7a8676c1afa444b88243ef4c576fb868\020\001\030\001 \001(\001" signature: "<redacted>""3\024\344\265Kj\366\225Hu`\222\225g\357;8\321%-\332H\234C\27357|\313\025\247\2556()\260\000\375\336\277`(\315\025\001>\316\330\302Q\'\341\0011\322_\353 \255W\306\252O0" signing_key_seq_num: 0 } } request_id { client_id: "5d282c7cc56c4638884f088c4a5046f1" seq_no: 352 first_incomplete_seq_no: 352 attempt_no: 0 }
I20240503 08:11:51.328141 29454 raft_consensus.cc:2258] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:11:51.330051 27428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 55760.5us
I20240503 08:11:51.336007 29454 ts_tablet_manager.cc:1875] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:11:51.339962 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 65990.3us
W20240503 08:11:51.343813 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 54003.1us
I20240503 08:11:51.365656 29454 ts_tablet_manager.cc:1888] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.115
I20240503 08:11:51.367161 29454 log.cc:1196] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 36fc7a106457453ba91f05b78a5d342b: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/08b4767b5b4f4fb0bfc73c35a408a5ac
I20240503 08:11:51.368628 26947 catalog_manager.cc:4830] TS 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): tablet 08b4767b5b4f4fb0bfc73c35a408a5ac (table rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]) successfully deleted
I20240503 08:11:51.405684 27490 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240503 08:11:51.459311 29293 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 114, Time since last communication: 0.000s
I20240503 08:11:51.485050 27494 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 110 to 116, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
W20240503 08:11:51.661124 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 44079us
W20240503 08:11:51.674666 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 45906.7us
W20240503 08:11:51.763988 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 102308us
W20240503 08:11:51.847733 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 63388.4us
W20240503 08:11:51.914228 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 53807.4us
I20240503 08:11:51.933149 29352 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap replayed 1/1 log segments. Stats: ops{read=117 overwritten=0 applied=117 ignored=0} inserts{seen=802 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:51.939574 29352 tablet_bootstrap.cc:492] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: Bootstrap complete.
I20240503 08:11:51.942829 29352 ts_tablet_manager.cc:1367] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: Time spent bootstrapping tablet: real 1.410s user 0.189s sys 0.079s
I20240503 08:11:51.943745 29352 raft_consensus.cc:348] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: true } }
I20240503 08:11:51.944175 29352 raft_consensus.cc:729] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Initialized, Role: LEARNER
I20240503 08:11:51.944451 29352 consensus_queue.cc:260] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 117, Last appended: 1.117, Last appended by leader: 117, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: true } }
I20240503 08:11:51.946245 29352 ts_tablet_manager.cc:1398] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:51.948926 27255 tablet_copy_service.cc:342] P 36fc7a106457453ba91f05b78a5d342b: Request end of tablet copy session f4d3aef6d5624cf4adc3859dbadccb80-8e4a8224c7c1430a9354e58bd16c7abd received from {username='slave'} at 127.24.123.1:39693
I20240503 08:11:51.949045 27255 tablet_copy_service.cc:433] P 36fc7a106457453ba91f05b78a5d342b: ending tablet copy session f4d3aef6d5624cf4adc3859dbadccb80-8e4a8224c7c1430a9354e58bd16c7abd on tablet 8e4a8224c7c1430a9354e58bd16c7abd with peer f4d3aef6d5624cf4adc3859dbadccb80
I20240503 08:11:52.082517 27098 raft_consensus.cc:1203] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.113->[1.114-1.120] Dedup: 1.117->[1.118-1.120]
W20240503 08:11:52.243885 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 163295us
I20240503 08:11:52.314375 29334 raft_consensus.cc:1050] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b: attempting to promote NON_VOTER f4d3aef6d5624cf4adc3859dbadccb80 to VOTER
W20240503 08:11:52.315023 29478 consensus_queue.cc:1159] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" responder_term: 1 status { last_received { term: 1 index: 121 } last_committed_idx: 121 last_received_current_leader { term: 1 index: 121 } }
I20240503 08:11:52.324355 29334 consensus_queue.cc:237] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.121, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } }
W20240503 08:11:52.325543 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 54817.2us
W20240503 08:11:52.326697 27559 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 55078.8us
W20240503 08:11:52.342800 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 68607.3us
I20240503 08:11:52.357530 27104 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEARNER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 08:11:52.358155 29475 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20240503 08:11:52.361398 27361 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 08:11:52.362365 29475 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20240503 08:11:52.362784 27753 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 08:11:52.363312 29334 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20240503 08:11:52.430569 29404 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Committing config change with OpId 1.122: config changed from index 114 to 122, f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:52.438040 27104 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 114 to 122, f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:52.441499 27102 consensus_queue.cc:237] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 135, Committed index: 135, Last appended: 1.135, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
W20240503 08:11:52.444299 27040 consensus_peers.cc:485] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c42929dcee3c461c8db91346ca1ab0f5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:52.449694 26955 catalog_manager.cc:5484] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: config changed from index 114 to 122, f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:52.448316 27751 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 114 to 122, f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:52.452180 27234 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240503 08:11:52.452843 29467 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I20240503 08:11:52.458009 27362 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240503 08:11:52.459111 29186 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I20240503 08:11:52.460013 27361 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 114 to 122, f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:52.464633 29186 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.136: config changed from index -1 to 136, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:52.465129 27362 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index -1 to 136, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:52.470554 26957 catalog_manager.cc:5484] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index -1 to 136, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:52.473436 27624 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 08:11:52.488123 27624 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 135. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240503 08:11:52.488891 29186 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.001s
I20240503 08:11:52.491698 27624 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index -1 to 136, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:52.494305 29334 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240503 08:11:52.498055 27621 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 114 to 122, f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:52.485285 27234 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index -1 to 136, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:52.538992 27490 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 136. (index mismatch)
I20240503 08:11:52.593593 29467 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 133, Time since last communication: 0.000s
I20240503 08:11:52.599668 27489 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
W20240503 08:11:52.756790 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 44601.5us
I20240503 08:11:52.790982 29475 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240503 08:11:52.793840 27490 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.136: config changed from index -1 to 136, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:52.873437 27494 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 114 to 122, f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:52.953681 27235 consensus_queue.cc:237] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.126, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } }
I20240503 08:11:52.960819 27098 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 08:11:52.998381 27755 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240503 08:11:52.999213 29404 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 08:11:52.999665 27361 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 08:11:53.000799 29404 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 08:11:53.001051 29478 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 08:11:53.012089 27622 raft_consensus.cc:1261] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240503 08:11:53.014983 29404 consensus_queue.cc:1035] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 08:11:53.019116 29478 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Committing config change with OpId 1.127: config changed from index 122 to 127, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:53.027446 26947 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8e4a8224c7c1430a9354e58bd16c7abd with cas_config_opid_index 122: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:53.028445 27755 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 122 to 127, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:53.034785 27621 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 122 to 127, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:53.034653 26955 catalog_manager.cc:5484] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b reported cstate change: config changed from index 122 to 127, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:53.051788 27473 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4a8224c7c1430a9354e58bd16c7abd with delete_type TABLET_DATA_TOMBSTONED (TS 19fb518a6ede476f809c743054941e97 not found in new config with opid_index 127) from {username='slave'} at 127.0.0.1:56480
I20240503 08:11:53.084405 29519 ts_tablet_manager.cc:902] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:53.097041 27363 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 122 to 127, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:53.142464 29521 tablet_replica.cc:330] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97: stopping tablet replica
I20240503 08:11:53.153200 27104 raft_consensus.cc:2932] T 8e4a8224c7c1430a9354e58bd16c7abd P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 122 to 127, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } } }
I20240503 08:11:53.197629 29521 raft_consensus.cc:2229] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:53.205569 29521 raft_consensus.cc:2258] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:53.234728 29521 ts_tablet_manager.cc:1875] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:53.237996 27362 consensus_queue.cc:237] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 180, Committed index: 180, Last appended: 1.180, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:53.241590 27234 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 08:11:53.242251 28949 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240503 08:11:53.249846 27750 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 08:11:53.251178 29519 tablet_copy_client.cc:308] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39261
I20240503 08:11:53.251899 27124 tablet_copy_service.cc:140] P f4d3aef6d5624cf4adc3859dbadccb80: Received BeginTabletCopySession request for tablet c42929dcee3c461c8db91346ca1ab0f5 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:34495)
I20240503 08:11:53.252065 27124 tablet_copy_service.cc:161] P f4d3aef6d5624cf4adc3859dbadccb80: Beginning new tablet copy session on tablet c42929dcee3c461c8db91346ca1ab0f5 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:34495: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-c42929dcee3c461c8db91346ca1ab0f5
I20240503 08:11:53.254832 27124 tablet_copy_source_session.cc:205] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:53.245237 27494 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 08:11:53.260963 28949 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240503 08:11:53.261219 29519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42929dcee3c461c8db91346ca1ab0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:53.264969 29407 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.181: config changed from index -1 to 181, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:53.265386 27493 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index -1 to 181, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:53.269868 27234 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index -1 to 181, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:53.272792 26959 catalog_manager.cc:5484] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index -1 to 181, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:53.277637 27102 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 08:11:53.280409 29407 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240503 08:11:53.286561 27102 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index -1 to 181, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:53.298202 29526 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
W20240503 08:11:53.308943 27301 consensus_peers.cc:485] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccb3a8dde7ec4d318b0961c8bad8032c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:53.315394 27749 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index -1 to 181, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:53.398113 29521 ts_tablet_manager.cc:1888] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.125
I20240503 08:11:53.434656 29519 tablet_copy_client.cc:783] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:53.457384 29519 tablet_copy_client.cc:647] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:53.464704 29519 tablet_copy_client.cc:523] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:53.467856 29519 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:53.551211 29521 log.cc:1196] T 8e4a8224c7c1430a9354e58bd16c7abd P 19fb518a6ede476f809c743054941e97: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/8e4a8224c7c1430a9354e58bd16c7abd
I20240503 08:11:53.631498 26948 catalog_manager.cc:4830] TS 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395): tablet 8e4a8224c7c1430a9354e58bd16c7abd (table rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]) successfully deleted
I20240503 08:11:53.720391 27494 consensus_queue.cc:237] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 140, Committed index: 140, Last appended: 1.140, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:53.725870 27232 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 08:11:53.727020 29351 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 08:11:53.740370 27625 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 08:11:53.741530 27752 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 08:11:53.742516 29351 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 08:11:53.745816 29351 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 139, Time since last communication: 0.000s
W20240503 08:11:53.747117 27431 consensus_peers.cc:485] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03b4df9c9c0c4ec59b00338de1ae2920. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:53.751971 29409 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.141: config changed from index -1 to 141, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
W20240503 08:11:53.756096 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43185, user_credentials={real_user=slave}} blocked reactor thread for 119702us
I20240503 08:11:53.797787 27621 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index -1 to 141, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:53.800639 29542 ts_tablet_manager.cc:902] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:11:53.809196 26958 catalog_manager.cc:5484] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index -1 to 141, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:53.850247 27232 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index -1 to 141, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:53.950512 27753 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index -1 to 141, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:54.084245 27101 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
W20240503 08:11:54.102998 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 88814.8us
W20240503 08:11:54.223070 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43185, user_credentials={real_user=slave}} blocked reactor thread for 46506.6us
I20240503 08:11:54.228611 29542 tablet_copy_client.cc:308] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:43977
I20240503 08:11:54.269304 28981 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 08:11:54.311168 27385 tablet_copy_service.cc:140] P fe8fdcdebbff4703a924f83646531115: Received BeginTabletCopySession request for tablet ccb3a8dde7ec4d318b0961c8bad8032c from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:40565)
I20240503 08:11:54.311383 27385 tablet_copy_service.cc:161] P fe8fdcdebbff4703a924f83646531115: Beginning new tablet copy session on tablet ccb3a8dde7ec4d318b0961c8bad8032c from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:40565: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-ccb3a8dde7ec4d318b0961c8bad8032c
I20240503 08:11:54.314397 27385 tablet_copy_source_session.cc:205] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:54.321986 29542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb3a8dde7ec4d318b0961c8bad8032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.334137 29542 tablet_copy_client.cc:783] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:54.334496 29542 tablet_copy_client.cc:647] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:54.341992 27098 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index -1 to 141, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:54.379480 29568 ts_tablet_manager.cc:902] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
W20240503 08:11:54.445557 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 69655.6us
I20240503 08:11:54.462168 29542 tablet_copy_client.cc:523] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:54.482596 29542 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:54.507757 29519 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=143 overwritten=0 applied=143 ignored=0} inserts{seen=815 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:54.508558 29519 tablet_bootstrap.cc:492] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:54.509124 29519 ts_tablet_manager.cc:1367] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 1.041s user 0.185s sys 0.010s
I20240503 08:11:54.510339 29519 raft_consensus.cc:348] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:54.510907 29519 raft_consensus.cc:729] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:11:54.523164 29519 consensus_queue.cc:260] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 143, Last appended: 1.143, Last appended by leader: 143, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 136 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:54.538585 29519 ts_tablet_manager.cc:1398] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.029s user 0.003s sys 0.004s
I20240503 08:11:54.550307 27124 tablet_copy_service.cc:342] P f4d3aef6d5624cf4adc3859dbadccb80: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-c42929dcee3c461c8db91346ca1ab0f5 received from {username='slave'} at 127.24.123.7:34495
I20240503 08:11:54.550463 27124 tablet_copy_service.cc:433] P f4d3aef6d5624cf4adc3859dbadccb80: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-c42929dcee3c461c8db91346ca1ab0f5 on tablet c42929dcee3c461c8db91346ca1ab0f5 with peer 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:54.609282 28749 raft_consensus.cc:1203] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.135->[1.136-1.153] Dedup: 1.143->[1.144-1.153]
I20240503 08:11:54.623168 27621 consensus_queue.cc:237] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.139, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
W20240503 08:11:54.624181 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 73214.7us
I20240503 08:11:54.627763 27231 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 08:11:54.628505 29505 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 08:11:54.628950 27365 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 08:11:54.629035 27750 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 08:11:54.629470 29505 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 08:11:54.630080 29505 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
W20240503 08:11:54.630493 27561 consensus_peers.cc:485] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a61402b1e1644ab7a0769b74e40e479b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:54.632383 29505 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.140: config changed from index -1 to 140, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:54.635509 27362 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index -1 to 140, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:54.636499 27750 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index -1 to 140, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:54.642174 29467 raft_consensus.cc:1050] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:54.643014 29467 consensus_queue.cc:237] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 1.153, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:54.649919 26955 catalog_manager.cc:5484] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed reported cstate change: config changed from index -1 to 140, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:54.654675 27232 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index -1 to 140, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:54.660702 28749 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240503 08:11:54.661340 29310 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240503 08:11:54.680122 27102 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
W20240503 08:11:54.685611 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 61158.5us
I20240503 08:11:54.711299 29568 tablet_copy_client.cc:308] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:41395
I20240503 08:11:54.712595 27515 tablet_copy_service.cc:140] P 19fb518a6ede476f809c743054941e97: Received BeginTabletCopySession request for tablet 03b4df9c9c0c4ec59b00338de1ae2920 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:60417)
I20240503 08:11:54.712759 27515 tablet_copy_service.cc:161] P 19fb518a6ede476f809c743054941e97: Beginning new tablet copy session on tablet 03b4df9c9c0c4ec59b00338de1ae2920 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:60417: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-03b4df9c9c0c4ec59b00338de1ae2920
I20240503 08:11:54.717620 27515 tablet_copy_source_session.cc:205] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:54.719357 29568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b4df9c9c0c4ec59b00338de1ae2920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:54.721628 27362 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240503 08:11:54.723569 27232 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240503 08:11:54.733611 29420 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 152, Time since last communication: 0.000s
I20240503 08:11:54.735956 29420 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 152, Time since last communication: 0.000s
W20240503 08:11:54.740370 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 101207us
I20240503 08:11:54.744128 27625 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240503 08:11:54.744599 29589 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 08:11:54.746702 29250 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240503 08:11:54.748883 27489 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240503 08:11:54.776302 29420 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.154: config changed from index 136 to 154, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:54.781834 28751 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 136 to 154, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:54.787382 29568 tablet_copy_client.cc:783] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:54.787716 29568 tablet_copy_client.cc:647] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:54.788547 26955 catalog_manager.cc:5484] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 reported cstate change: config changed from index 136 to 154, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:54.791782 27233 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 136 to 154, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:54.793756 27104 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index -1 to 140, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:54.793725 27363 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 136 to 154, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:54.806902 29586 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 152, Time since last communication: 0.000s
I20240503 08:11:54.816872 27625 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 136 to 154, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:54.818693 27494 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 136 to 154, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:54.856141 29568 tablet_copy_client.cc:523] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:54.870770 29568 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:54.889039 27102 consensus_queue.cc:237] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 1.158, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:54.902653 27362 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 08:11:54.907554 27231 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 08:11:54.946086 27493 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 08:11:54.891705 28751 raft_consensus.cc:1261] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 08:11:54.958029 29420 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 157, Time since last communication: 0.002s
I20240503 08:11:54.960143 29583 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 08:11:54.991077 29583 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 08:11:54.995476 29420 consensus_queue.cc:1035] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 08:11:55.007625 29519 ts_tablet_manager.cc:902] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:11:55.096757 29420 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.159: config changed from index 154 to 159, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:55.222600 26950 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c42929dcee3c461c8db91346ca1ab0f5 with cas_config_opid_index 154: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:55.223919 28748 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 154 to 159, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:55.231580 26958 catalog_manager.cc:5484] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 reported cstate change: config changed from index 154 to 159, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:55.235487 27605 tablet_service.cc:1514] Processing DeleteTablet for tablet c42929dcee3c461c8db91346ca1ab0f5 with delete_type TABLET_DATA_TOMBSTONED (TS 2de33eb4315b4547889d1e6edae2f2dd not found in new config with opid_index 159) from {username='slave'} at 127.0.0.1:33064
I20240503 08:11:55.237143 29618 tablet_replica.cc:330] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: stopping tablet replica
I20240503 08:11:55.237344 29618 raft_consensus.cc:2229] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:55.240408 29618 pending_rounds.cc:62] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: Trying to abort 1 pending ops.
I20240503 08:11:55.240525 29618 pending_rounds.cc:68] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: Aborting op as it isn't in flight: id { term: 1 index: 158 } timestamp: 7023509155235135488 op_type: WRITE_OP write_request { tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000?\312}SW!\343\244\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""017a1ce56ffdfe49" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509155082387456 authz_token { token_data: "\010\345\272\322\261\006\"3\n\005slave\022*\n 7a8676c1afa444b88243ef4c576fb868\020\001\030\001 \001(\001" signature: "<redacted>""3\024\344\265Kj\366\225Hu`\222\225g\357;8\321%-\332H\234C\27357|\313\025\247\2556()\260\000\375\336\277`(\315\025\001>\316\330\302Q\'\341\0011\322_\353 \255W\306\252O0" signing_key_seq_num: 0 } } request_id { client_id: "5d282c7cc56c4638884f088c4a5046f1" seq_no: 438 first_incomplete_seq_no: 438 attempt_no: 0 }
I20240503 08:11:55.241268 29618 raft_consensus.cc:2258] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:55.243619 29618 ts_tablet_manager.cc:1875] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:11:55.249832 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 45099.1us
I20240503 08:11:55.273985 27232 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 154 to 159, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:55.286417 27362 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 154 to 159, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:55.291865 29618 ts_tablet_manager.cc:1888] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.158
I20240503 08:11:55.292428 29618 log.cc:1196] T c42929dcee3c461c8db91346ca1ab0f5 P 2de33eb4315b4547889d1e6edae2f2dd: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/wals/c42929dcee3c461c8db91346ca1ab0f5
I20240503 08:11:55.299149 26948 catalog_manager.cc:4830] TS 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185): tablet c42929dcee3c461c8db91346ca1ab0f5 (table rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]) successfully deleted
I20240503 08:11:55.311208 27494 raft_consensus.cc:2932] T c42929dcee3c461c8db91346ca1ab0f5 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 154 to 159, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:55.329867 29519 tablet_copy_client.cc:308] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43185
I20240503 08:11:55.330842 27645 tablet_copy_service.cc:140] P 2de33eb4315b4547889d1e6edae2f2dd: Received BeginTabletCopySession request for tablet a61402b1e1644ab7a0769b74e40e479b from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:42183)
I20240503 08:11:55.331063 27645 tablet_copy_service.cc:161] P 2de33eb4315b4547889d1e6edae2f2dd: Beginning new tablet copy session on tablet a61402b1e1644ab7a0769b74e40e479b from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:42183: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-a61402b1e1644ab7a0769b74e40e479b
I20240503 08:11:55.333896 27645 tablet_copy_source_session.cc:205] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:55.336722 29519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61402b1e1644ab7a0769b74e40e479b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:55.355330 29519 tablet_copy_client.cc:783] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:55.355571 29519 tablet_copy_client.cc:647] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:55.361411 29519 tablet_copy_client.cc:523] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:55.365715 29519 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
W20240503 08:11:55.487241 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 57454.1us
I20240503 08:11:55.725929 29542 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=196 overwritten=0 applied=196 ignored=0} inserts{seen=881 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:55.730562 29542 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:55.730908 29542 ts_tablet_manager.cc:1367] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 1.248s user 0.247s sys 0.018s
I20240503 08:11:55.731739 29542 raft_consensus.cc:348] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:55.732376 29542 raft_consensus.cc:729] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:11:55.732585 29542 consensus_queue.cc:260] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 196, Last appended: 1.196, Last appended by leader: 196, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:55.743181 29542 ts_tablet_manager.cc:1398] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.012s user 0.000s sys 0.004s
I20240503 08:11:55.795945 29519 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=144 overwritten=0 applied=144 ignored=0} inserts{seen=775 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:55.796386 29519 tablet_bootstrap.cc:492] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:55.796671 29519 ts_tablet_manager.cc:1367] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 0.431s user 0.181s sys 0.014s
I20240503 08:11:55.798105 29519 raft_consensus.cc:348] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:55.798385 29519 raft_consensus.cc:729] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:11:55.799818 29519 consensus_queue.cc:260] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 144, Last appended: 1.144, Last appended by leader: 144, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:55.802647 29519 ts_tablet_manager.cc:1398] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:11:55.805016 27645 tablet_copy_service.cc:342] P 2de33eb4315b4547889d1e6edae2f2dd: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-a61402b1e1644ab7a0769b74e40e479b received from {username='slave'} at 127.24.123.7:42183
I20240503 08:11:55.805141 27645 tablet_copy_service.cc:433] P 2de33eb4315b4547889d1e6edae2f2dd: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-a61402b1e1644ab7a0769b74e40e479b on tablet a61402b1e1644ab7a0769b74e40e479b with peer 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:55.880502 27385 tablet_copy_service.cc:342] P fe8fdcdebbff4703a924f83646531115: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-ccb3a8dde7ec4d318b0961c8bad8032c received from {username='slave'} at 127.24.123.7:40565
I20240503 08:11:55.880659 27385 tablet_copy_service.cc:433] P fe8fdcdebbff4703a924f83646531115: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-ccb3a8dde7ec4d318b0961c8bad8032c on tablet ccb3a8dde7ec4d318b0961c8bad8032c with peer 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:55.966851 28749 raft_consensus.cc:1203] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.139->[1.140-1.155] Dedup: 1.144->[1.145-1.155]
I20240503 08:11:56.155230 29630 raft_consensus.cc:1050] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:56.156340 29630 consensus_queue.cc:237] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 1.157, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:56.159180 27235 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 08:11:56.159658 29589 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 08:11:56.162659 28751 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 08:11:56.163589 27362 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 08:11:56.164496 29630 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.001s
I20240503 08:11:56.169059 29505 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 08:11:56.179313 27753 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 08:11:56.179875 29361 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 08:11:56.189525 29361 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.158: config changed from index 140 to 158, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.205183 27361 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 140 to 158, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.205235 28748 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 140 to 158, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.206634 27749 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 140 to 158, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.208271 26958 catalog_manager.cc:5484] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index 140 to 158, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:56.212626 27234 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 140 to 158, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.219403 27102 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 08:11:56.224668 29361 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 08:11:56.235564 27102 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 140 to 158, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
W20240503 08:11:56.270026 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 136854us
I20240503 08:11:56.337894 28749 raft_consensus.cc:1203] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.180->[1.181-1.211] Dedup: 1.196->[1.197-1.211]
W20240503 08:11:56.364490 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 57044.3us
I20240503 08:11:56.403065 29568 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=142 overwritten=0 applied=142 ignored=0} inserts{seen=814 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:56.403586 29568 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:11:56.404498 29568 ts_tablet_manager.cc:1367] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 1.534s user 0.187s sys 0.037s
I20240503 08:11:56.405280 29568 raft_consensus.cc:348] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:56.405663 29568 raft_consensus.cc:729] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:11:56.405951 29568 consensus_queue.cc:260] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 142, Last appended: 1.142, Last appended by leader: 142, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:56.407450 29568 ts_tablet_manager.cc:1398] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:11:56.408129 27515 tablet_copy_service.cc:342] P 19fb518a6ede476f809c743054941e97: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-03b4df9c9c0c4ec59b00338de1ae2920 received from {username='slave'} at 127.24.123.7:60417
I20240503 08:11:56.408257 27515 tablet_copy_service.cc:433] P 19fb518a6ede476f809c743054941e97: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-03b4df9c9c0c4ec59b00338de1ae2920 on tablet 03b4df9c9c0c4ec59b00338de1ae2920 with peer 1980d01e2e3e4840a8ddfca4d4e366b2
I20240503 08:11:56.462766 29575 raft_consensus.cc:1050] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:56.464952 29575 consensus_queue.cc:237] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 1.213, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:56.469748 28748 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 214. (index mismatch)
I20240503 08:11:56.470551 27750 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 213. Preceding OpId from leader: term: 1 index: 214. (index mismatch)
I20240503 08:11:56.470953 27490 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 213. Preceding OpId from leader: term: 1 index: 214. (index mismatch)
I20240503 08:11:56.471624 29533 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 212, Time since last communication: 0.000s
I20240503 08:11:56.472047 29585 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20240503 08:11:56.475147 29533 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20240503 08:11:56.484246 29585 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.214: config changed from index 181 to 214, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.491139 27494 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.214: config changed from index 181 to 214, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.496412 26959 catalog_manager.cc:5484] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 181 to 214, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:56.498766 28748 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.214: config changed from index 181 to 214, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.501680 27750 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.214: config changed from index 181 to 214, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.519124 27103 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 214. (index mismatch)
I20240503 08:11:56.520380 28751 raft_consensus.cc:1203] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.212->[1.213-1.213] Dedup: 1.213->[]
I20240503 08:11:56.538009 27230 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 213. Preceding OpId from leader: term: 1 index: 214. (index mismatch)
I20240503 08:11:56.554426 29386 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 212, Time since last communication: 0.000s
W20240503 08:11:56.592554 27299 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43185, user_credentials={real_user=slave}} blocked reactor thread for 54955.1us
W20240503 08:11:56.596006 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 54921.9us
I20240503 08:11:56.604156 29533 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 08:11:56.624482 27098 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.214: config changed from index 181 to 214, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
W20240503 08:11:56.659046 27298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 117650us
I20240503 08:11:56.664770 27228 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.214: config changed from index 181 to 214, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.748584 28747 raft_consensus.cc:1203] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.140->[1.141-1.166] Dedup: 1.142->[1.143-1.166]
I20240503 08:11:56.811815 27624 consensus_queue.cc:237] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 159, Committed index: 159, Last appended: 1.160, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:56.815656 27360 consensus_queue.cc:237] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 1.215, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:56.818414 28748 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20240503 08:11:56.818444 27753 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20240503 08:11:56.819002 29533 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20240503 08:11:56.819618 29659 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20240503 08:11:56.821205 28748 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 08:11:56.823956 29533 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.216: config changed from index 214 to 216, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.824533 27753 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 214 to 216, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.831247 27749 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 08:11:56.834364 27490 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20240503 08:11:56.835211 27360 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 08:11:56.837967 29659 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20240503 08:11:56.837874 26959 catalog_manager.cc:5484] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed reported cstate change: config changed from index 214 to 216, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.839030 28748 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 214 to 216, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.841738 27213 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb3a8dde7ec4d318b0961c8bad8032c with delete_type TABLET_DATA_TOMBSTONED (TS 36fc7a106457453ba91f05b78a5d342b not found in new config with opid_index 216) from {username='slave'} at 127.0.0.1:50738
I20240503 08:11:56.834843 29483 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 159, Time since last communication: 0.017s
I20240503 08:11:56.842900 29361 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 08:11:56.849298 27104 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20240503 08:11:56.854028 29386 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20240503 08:11:56.855607 29361 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 08:11:56.865717 27750 raft_consensus.cc:1203] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.159->[1.160-1.161] Dedup: 1.160->[1.161-1.161]
I20240503 08:11:56.868016 26947 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccb3a8dde7ec4d318b0961c8bad8032c with cas_config_opid_index 214: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:56.869427 27490 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 214 to 216, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.871944 29361 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.872797 27360 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.877473 28981 raft_consensus.cc:1050] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:11:56.878186 28981 consensus_queue.cc:237] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 1.167, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:56.883671 28748 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 08:11:56.884784 29650 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 08:11:56.884967 27753 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 08:11:56.891155 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a61402b1e1644ab7a0769b74e40e479b with cas_config_opid_index 158: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:56.893911 27104 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 214 to 216, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.896853 26960 catalog_manager.cc:5484] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index 158 to 161, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:56.899029 28748 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.902515 27215 tablet_service.cc:1514] Processing DeleteTablet for tablet a61402b1e1644ab7a0769b74e40e479b with delete_type TABLET_DATA_TOMBSTONED (TS 36fc7a106457453ba91f05b78a5d342b not found in new config with opid_index 161) from {username='slave'} at 127.0.0.1:50738
I20240503 08:11:56.904341 27101 raft_consensus.cc:1261] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 08:11:56.912972 27104 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 08:11:56.913939 29483 consensus_queue.cc:1035] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 08:11:56.919241 27101 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.935711 27624 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 08:11:56.942052 29580 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 08:11:56.943936 29580 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 08:11:56.945165 29580 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 08:11:56.954996 27748 raft_consensus.cc:2932] T a61402b1e1644ab7a0769b74e40e479b P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:56.960536 27098 raft_consensus.cc:1203] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.166->[1.167-1.168] Dedup: 1.167->[1.168-1.168]
I20240503 08:11:56.989595 29540 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.168: config changed from index 141 to 168, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.017465 27749 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 141 to 168, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.032903 27102 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 141 to 168, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.039181 26959 catalog_manager.cc:5484] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: config changed from index 141 to 168, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.050782 29669 tablet_replica.cc:330] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b: stopping tablet replica
I20240503 08:11:57.056860 28751 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 141 to 168, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.055600 27234 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 08:11:57.068698 29650 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 08:11:57.072436 27228 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 141 to 168, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.083143 27625 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 141 to 168, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.085647 29669 raft_consensus.cc:2229] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 08:11:57.105237 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 53008.7us
I20240503 08:11:57.109786 29669 raft_consensus.cc:2258] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:57.122982 27494 consensus_queue.cc:237] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 1.170, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:11:57.125998 27749 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 08:11:57.126605 29540 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 08:11:57.128799 27234 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 08:11:57.129345 29673 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 08:11:57.132606 28747 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 08:11:57.151820 29673 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 168, Time since last communication: 0.001s
I20240503 08:11:57.156212 28751 raft_consensus.cc:1203] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.168->[1.169-1.169] Dedup: 1.169->[]
I20240503 08:11:57.157757 29669 ts_tablet_manager.cc:1875] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:57.157488 28751 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.162318 28981 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.166853 26960 catalog_manager.cc:5484] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 reported cstate change: config changed from index 168 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.170570 27084 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b4df9c9c0c4ec59b00338de1ae2920 with delete_type TABLET_DATA_TOMBSTONED (TS f4d3aef6d5624cf4adc3859dbadccb80 not found in new config with opid_index 171) from {username='slave'} at 127.0.0.1:57128
I20240503 08:11:57.187482 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03b4df9c9c0c4ec59b00338de1ae2920 with cas_config_opid_index 168: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:57.197860 27750 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.207096 29685 tablet_replica.cc:330] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80: stopping tablet replica
I20240503 08:11:57.210228 29685 raft_consensus.cc:2229] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:57.211119 27234 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.237865 29685 raft_consensus.cc:2258] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:57.239338 29685 ts_tablet_manager.cc:1875] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:11:57.238726 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 53207.2us
I20240503 08:11:57.255692 29685 ts_tablet_manager.cc:1888] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.169
I20240503 08:11:57.258349 29685 log.cc:1196] T 03b4df9c9c0c4ec59b00338de1ae2920 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/03b4df9c9c0c4ec59b00338de1ae2920
I20240503 08:11:57.264225 26950 catalog_manager.cc:4830] TS f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): tablet 03b4df9c9c0c4ec59b00338de1ae2920 (table rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]) successfully deleted
I20240503 08:11:57.270437 27622 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 08:11:57.280504 29650 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 168, Time since last communication: 0.000s
I20240503 08:11:57.314947 27625 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:11:57.423575 29669 ts_tablet_manager.cc:1888] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.215
I20240503 08:11:57.430711 29669 log.cc:1196] T ccb3a8dde7ec4d318b0961c8bad8032c P 36fc7a106457453ba91f05b78a5d342b: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/ccb3a8dde7ec4d318b0961c8bad8032c
I20240503 08:11:57.514633 26947 catalog_manager.cc:4830] TS 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): tablet ccb3a8dde7ec4d318b0961c8bad8032c (table rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]) successfully deleted
I20240503 08:11:57.522053 29669 tablet_replica.cc:330] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: stopping tablet replica
I20240503 08:11:57.524312 29669 raft_consensus.cc:2229] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:57.542475 29669 pending_rounds.cc:62] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: Trying to abort 1 pending ops.
I20240503 08:11:57.543087 29669 pending_rounds.cc:68] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: Aborting op as it isn't in flight: id { term: 1 index: 160 } timestamp: 7023509163225387008 op_type: WRITE_OP write_request { tablet_id: "a61402b1e1644ab7a0769b74e40e479b" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\265\202\223X#\276o\355\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c37686f7352d87cf" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509162167271424 authz_token { token_data: "\010\346\272\322\261\006\"3\n\005slave\022*\n 225aa297be6640d8b676ce90a19a1ba4\020\001\030\001 \001(\001" signature: "<redacted>""\220\n\035\224u>\361\nE\372H*}G.\305\'\377\306UD\241\317TY\036\007\342V\376\020\006>\n\266<\037\227\345jYq\230m\305\r\231\261\337j\352\021Va\332\tLd\376j\327C\010(" signing_key_seq_num: 0 } } request_id { client_id: "d0c59b8a52504a45985c7239df847fe0" seq_no: 453 first_incomplete_seq_no: 453 attempt_no: 0 }
I20240503 08:11:57.546581 29669 raft_consensus.cc:2258] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:11:57.663272 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 101085us
I20240503 08:11:57.663794 29669 ts_tablet_manager.cc:1875] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:57.766808 29669 ts_tablet_manager.cc:1888] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.160
I20240503 08:11:57.767014 29669 log.cc:1196] T a61402b1e1644ab7a0769b74e40e479b P 36fc7a106457453ba91f05b78a5d342b: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/a61402b1e1644ab7a0769b74e40e479b
I20240503 08:11:57.767941 26947 catalog_manager.cc:4830] TS 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): tablet a61402b1e1644ab7a0769b74e40e479b (table rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]) successfully deleted
I20240503 08:11:57.906291 27104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5"
dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42708
I20240503 08:11:57.906654 27104 raft_consensus.cc:595] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 LEADER]: Received request to transfer leadership
I20240503 08:11:57.962117 29250 raft_consensus.cc:979] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: : Instructing follower 36fc7a106457453ba91f05b78a5d342b to start an election
I20240503 08:11:57.962301 29250 raft_consensus.cc:1067] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 LEADER]: Signalling peer 36fc7a106457453ba91f05b78a5d342b to start an election
I20240503 08:11:57.963654 27235 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5"
dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
from {username='slave'} at 127.24.123.1:39693
I20240503 08:11:57.964699 27235 raft_consensus.cc:482] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:57.964843 27235 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:57.973543 27235 raft_consensus.cc:504] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } }
I20240503 08:11:57.974793 27360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:11:57.975008 27360 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:57.989364 27360 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 3.
I20240503 08:11:57.995060 28748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2"
I20240503 08:11:57.995395 28748 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:57.995451 27102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:11:57.995625 27102 raft_consensus.cc:3032] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:11:57.995719 27102 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f4d3aef6d5624cf4adc3859dbadccb80, State: Running, Role: LEADER
I20240503 08:11:57.996027 27102 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } }
I20240503 08:11:57.996557 27102 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:57.998330 28748 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 3.
I20240503 08:11:57.999380 27102 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 3.
I20240503 08:11:58.000032 27235 leader_election.cc:288] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 3 election: Requested vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167)
I20240503 08:11:58.003911 27171 leader_election.cc:302] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2, 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:11:58.004205 29642 raft_consensus.cc:2781] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:11:58.004317 29642 raft_consensus.cc:686] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 LEADER]: Becoming Leader. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Running, Role: LEADER
I20240503 08:11:58.004607 29642 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } }
I20240503 08:11:58.034021 26960 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: term changed from 2 to 3, leader changed from f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) to 36fc7a106457453ba91f05b78a5d342b (127.24.123.2). New cstate: current_term: 3 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:11:58.040263 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 62779.5us
I20240503 08:11:58.055620 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:11:58.055845 27624 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:58.063347 27489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "36fc7a106457453ba91f05b78a5d342b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:11:58.063655 27489 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:58.066464 27489 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 3.
I20240503 08:11:58.098558 27624 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fc7a106457453ba91f05b78a5d342b in term 3.
W20240503 08:11:58.297426 27169 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.24.123.62:40087, user_credentials={real_user=slave}} blocked reactor thread for 44890.7us
I20240503 08:11:58.398020 27104 consensus_queue.cc:237] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 164, Committed index: 164, Last appended: 1.165, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:11:58.405454 27360 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch)
I20240503 08:11:58.406247 29467 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 164, Time since last communication: 0.000s
W20240503 08:11:58.407204 27040 consensus_peers.cc:485] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): Couldn't send request to peer 1980d01e2e3e4840a8ddfca4d4e366b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f94c738c0f34d04bd933cc09c05d6a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:58.414474 27749 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch)
I20240503 08:11:58.415349 29250 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 08:11:58.429241 29586 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.166: config changed from index -1 to 166, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:58.437317 27360 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index -1 to 166, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:58.442916 27231 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 166. (index mismatch)
I20240503 08:11:58.473019 27750 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index -1 to 166, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:58.473248 26959 catalog_manager.cc:5484] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: config changed from index -1 to 166, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:58.475862 28747 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:11:58.476696 29481 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:58.539358 29583 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 08:11:58.544809 27235 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index -1 to 166, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:58.551738 27494 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 162. Preceding OpId from leader: term: 1 index: 166. (index mismatch)
I20240503 08:11:58.562700 27624 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:11:58.576911 27102 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:11:58.577463 29642 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:58.585492 29642 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:58.593495 29731 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 162, Time since last communication: 0.000s
I20240503 08:11:58.594635 27489 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:11:58.626530 27365 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:11:58.627820 29404 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:58.638617 27490 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index -1 to 166, NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) added. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } } }
I20240503 08:11:58.639485 29404 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:11:58.813747 27169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 145936us
W20240503 08:11:58.814798 27171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 141767us
I20240503 08:11:58.902856 29765 ts_tablet_manager.cc:902] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: Initiating tablet copy from peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:11:59.146903 29765 tablet_copy_client.cc:308] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39261
I20240503 08:11:59.147667 27124 tablet_copy_service.cc:140] P f4d3aef6d5624cf4adc3859dbadccb80: Received BeginTabletCopySession request for tablet 8f94c738c0f34d04bd933cc09c05d6a6 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 ({username='slave'} at 127.24.123.7:34495)
I20240503 08:11:59.147807 27124 tablet_copy_service.cc:161] P f4d3aef6d5624cf4adc3859dbadccb80: Beginning new tablet copy session on tablet 8f94c738c0f34d04bd933cc09c05d6a6 from peer 1980d01e2e3e4840a8ddfca4d4e366b2 at {username='slave'} at 127.24.123.7:34495: session id = 1980d01e2e3e4840a8ddfca4d4e366b2-8f94c738c0f34d04bd933cc09c05d6a6
I20240503 08:11:59.152186 27124 tablet_copy_source_session.cc:205] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 08:11:59.201862 27169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 215357us
W20240503 08:11:59.206420 27171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 186483us
I20240503 08:11:59.245141 27231 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } }
I20240503 08:11:59.258795 28749 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:11:59.261813 27491 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:11:59.262424 29404 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:59.280925 29404 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:59.285965 27625 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:11:59.293892 29404 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:59.294023 29765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f94c738c0f34d04bd933cc09c05d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:59.294380 29642 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:59.298986 26947 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8d5c3e2f8e6428aa457ba21d536cea5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:59.299690 27490 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
W20240503 08:11:59.304656 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 108385us
I20240503 08:11:59.305222 28749 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:59.307296 27362 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:11:59.308255 26959 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: config changed from index 3 to 5, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New cstate: current_term: 3 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:59.309705 29642 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:59.312692 27360 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:59.326412 27625 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:11:59.375101 29765 tablet_copy_client.cc:783] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:59.379801 29765 tablet_copy_client.cc:647] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:59.392318 29765 tablet_copy_client.cc:523] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:59.401396 29765 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap starting.
I20240503 08:11:59.403868 27084 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d5c3e2f8e6428aa457ba21d536cea5 with delete_type TABLET_DATA_TOMBSTONED (TS f4d3aef6d5624cf4adc3859dbadccb80 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57128
I20240503 08:11:59.446909 29792 tablet_replica.cc:330] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: stopping tablet replica
I20240503 08:11:59.454062 29792 raft_consensus.cc:2229] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:59.457391 29792 raft_consensus.cc:2258] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:59.473922 29792 ts_tablet_manager.cc:1875] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:59.521858 29792 ts_tablet_manager.cc:1888] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240503 08:11:59.530810 29792 log.cc:1196] T b8d5c3e2f8e6428aa457ba21d536cea5 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8d5c3e2f8e6428aa457ba21d536cea5
I20240503 08:11:59.549232 26950 catalog_manager.cc:4830] TS f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): tablet b8d5c3e2f8e6428aa457ba21d536cea5 (table TestTable [id=0008359f803b498b9c188b0fffbee30e]) successfully deleted
W20240503 08:11:59.594547 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 69989.1us
I20240503 08:12:00.110864 29765 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=174 overwritten=0 applied=174 ignored=0} inserts{seen=859 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:00.111791 29765 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: Bootstrap complete.
I20240503 08:12:00.112303 29765 ts_tablet_manager.cc:1367] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent bootstrapping tablet: real 0.711s user 0.222s sys 0.010s
I20240503 08:12:00.113104 29765 raft_consensus.cc:348] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:12:00.113492 29765 raft_consensus.cc:729] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1980d01e2e3e4840a8ddfca4d4e366b2, State: Initialized, Role: LEARNER
I20240503 08:12:00.113740 29765 consensus_queue.cc:260] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 174, Last appended: 1.174, Last appended by leader: 174, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: true } }
I20240503 08:12:00.122740 29765 ts_tablet_manager.cc:1398] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20240503 08:12:00.123749 27124 tablet_copy_service.cc:342] P f4d3aef6d5624cf4adc3859dbadccb80: Request end of tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-8f94c738c0f34d04bd933cc09c05d6a6 received from {username='slave'} at 127.24.123.7:34495
I20240503 08:12:00.123873 27124 tablet_copy_service.cc:433] P f4d3aef6d5624cf4adc3859dbadccb80: ending tablet copy session 1980d01e2e3e4840a8ddfca4d4e366b2-8f94c738c0f34d04bd933cc09c05d6a6 on tablet 8f94c738c0f34d04bd933cc09c05d6a6 with peer 1980d01e2e3e4840a8ddfca4d4e366b2
W20240503 08:12:00.196401 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 48971.1us
I20240503 08:12:00.373322 28747 raft_consensus.cc:1203] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.165->[1.166-1.177] Dedup: 1.174->[1.175-1.177]
I20240503 08:12:00.627848 29782 raft_consensus.cc:1050] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80: attempting to promote NON_VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 to VOTER
I20240503 08:12:00.628744 29782 consensus_queue.cc:237] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 1.180, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:12:00.630559 29831 raft_consensus.cc:1013] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: attempt to promote peer 1980d01e2e3e4840a8ddfca4d4e366b2: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:12:00.634366 27360 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 08:12:00.634898 29586 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 179, Time since last communication: 0.000s
W20240503 08:12:00.638324 27169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 131102us
I20240503 08:12:00.640075 27234 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 08:12:00.648427 29583 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 178, Time since last communication: 0.000s
I20240503 08:12:00.689513 27490 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 08:12:00.690655 28749 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 LEARNER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 08:12:00.691242 29837 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 179, Time since last communication: 0.000s
I20240503 08:12:00.711234 27752 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240503 08:12:00.713990 29832 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 179, Time since last communication: 0.000s
I20240503 08:12:00.715374 29583 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 179, Time since last communication: 0.000s
I20240503 08:12:00.723387 29583 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.181: config changed from index 166 to 181, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:00.730548 26958 catalog_manager.cc:5484] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: config changed from index 166 to 181, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:00.733265 28751 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 166 to 181, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:00.733742 27752 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 166 to 181, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:00.739929 27361 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 166 to 181, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:00.823854 27234 raft_consensus.cc:1203] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.178->[1.179-1.181] Dedup: 1.179->[1.180-1.181]
I20240503 08:12:00.844493 27491 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 166 to 181, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:00.875854 27234 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 166 to 181, 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:01.073549 27101 consensus_queue.cc:237] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 188, Committed index: 188, Last appended: 1.188, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:12:01.076808 28750 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 188. Preceding OpId from leader: term: 1 index: 189. (index mismatch)
I20240503 08:12:01.077389 29583 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s
I20240503 08:12:01.078892 27748 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 188. Preceding OpId from leader: term: 1 index: 189. (index mismatch)
I20240503 08:12:01.079427 29583 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s
I20240503 08:12:01.082937 29839 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.189: config changed from index 181 to 189, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:01.083485 27748 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.189: config changed from index 181 to 189, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:01.086892 26950 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f94c738c0f34d04bd933cc09c05d6a6 with cas_config_opid_index 181: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:01.092545 26961 catalog_manager.cc:5484] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: config changed from index 181 to 189, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:01.097141 27213 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f94c738c0f34d04bd933cc09c05d6a6 with delete_type TABLET_DATA_TOMBSTONED (TS 36fc7a106457453ba91f05b78a5d342b not found in new config with opid_index 189) from {username='slave'} at 127.0.0.1:50738
I20240503 08:12:01.100324 28750 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.189: config changed from index 181 to 189, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:01.108496 27360 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 188. Preceding OpId from leader: term: 1 index: 189. (index mismatch)
I20240503 08:12:01.109036 29839 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s
I20240503 08:12:01.110374 27365 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.189: config changed from index 181 to 189, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:01.116250 29862 tablet_replica.cc:330] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b: stopping tablet replica
I20240503 08:12:01.116539 29862 raft_consensus.cc:2229] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:01.116902 29862 raft_consensus.cc:2258] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:01.120229 29862 ts_tablet_manager.cc:1875] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:01.131006 27494 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 188. Preceding OpId from leader: term: 1 index: 189. (index mismatch)
I20240503 08:12:01.134960 29832 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s
I20240503 08:12:01.138384 27494 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.189: config changed from index 181 to 189, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } }
I20240503 08:12:01.140120 29862 ts_tablet_manager.cc:1888] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.188
I20240503 08:12:01.140244 29862 log.cc:1196] T 8f94c738c0f34d04bd933cc09c05d6a6 P 36fc7a106457453ba91f05b78a5d342b: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/8f94c738c0f34d04bd933cc09c05d6a6
I20240503 08:12:01.141007 26947 catalog_manager.cc:4830] TS 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): tablet 8f94c738c0f34d04bd933cc09c05d6a6 (table rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]) successfully deleted
W20240503 08:12:01.236105 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 94868.4us
W20240503 08:12:01.701030 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 62839us
W20240503 08:12:02.839789 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 56141.8us
W20240503 08:12:03.149076 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 45591.7us
W20240503 08:12:03.164033 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 309613us
I20240503 08:12:03.195941 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:03.402101 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:12:03.405908 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 100707us
I20240503 08:12:03.504304 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:03.740609 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:03.777438 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:03.797902 27213 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:12:04.522192 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 48004.6us
I20240503 08:12:04.569027 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:05.234629 27494 consensus_queue.cc:237] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 159, Committed index: 159, Last appended: 1.159, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 160 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } }
W20240503 08:12:05.239045 27428 consensus_peers.cc:485] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 -> Peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Couldn't send request to peer 36fc7a106457453ba91f05b78a5d342b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f09f0689f0714d8c8f45e24bd9adb533. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:05.240543 27624 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 1 index: 160. (index mismatch)
I20240503 08:12:05.242098 29931 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 160, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 08:12:05.247457 28748 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 1 index: 160. (index mismatch)
I20240503 08:12:05.248368 27101 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 1 index: 160. (index mismatch)
I20240503 08:12:05.250206 29811 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 160, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 08:12:05.251343 29811 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 160, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 08:12:05.257709 27749 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 1 index: 160. (index mismatch)
I20240503 08:12:05.270545 28981 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 160, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 08:12:05.271035 29971 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.160: config changed from index 64 to 160, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 160 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
I20240503 08:12:05.277443 27624 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.160: config changed from index 64 to 160, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 160 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
I20240503 08:12:05.281690 26954 catalog_manager.cc:5484] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 reported cstate change: config changed from index 64 to 160, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 160 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.288475 28748 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.160: config changed from index 64 to 160, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 160 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
W20240503 08:12:05.304445 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 103655us
W20240503 08:12:05.307087 29874 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.024s sys 0.043s
W20240503 08:12:05.307372 29874 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.556s user 0.024s sys 0.043s
I20240503 08:12:05.307530 27101 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.160: config changed from index 64 to 160, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 160 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
I20240503 08:12:05.283557 27750 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.160: config changed from index 64 to 160, NON_VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) added. New config: { opid_index: 160 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } } }
W20240503 08:12:05.604090 27040 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 45184.3us
W20240503 08:12:05.663873 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 72652.1us
I20240503 08:12:05.887276 30039 ts_tablet_manager.cc:902] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: Initiating tablet copy from peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
W20240503 08:12:05.946542 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 91555.9us
W20240503 08:12:06.016522 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 69399.2us
W20240503 08:12:06.606597 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 184362us
I20240503 08:12:06.758884 30039 tablet_copy_client.cc:308] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:41395
I20240503 08:12:06.887495 27515 tablet_copy_service.cc:140] P 19fb518a6ede476f809c743054941e97: Received BeginTabletCopySession request for tablet f09f0689f0714d8c8f45e24bd9adb533 from peer 36fc7a106457453ba91f05b78a5d342b ({username='slave'} at 127.24.123.2:59799)
I20240503 08:12:06.887681 27515 tablet_copy_service.cc:161] P 19fb518a6ede476f809c743054941e97: Beginning new tablet copy session on tablet f09f0689f0714d8c8f45e24bd9adb533 from peer 36fc7a106457453ba91f05b78a5d342b at {username='slave'} at 127.24.123.2:59799: session id = 36fc7a106457453ba91f05b78a5d342b-f09f0689f0714d8c8f45e24bd9adb533
I20240503 08:12:06.890602 27515 tablet_copy_source_session.cc:205] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:06.901491 30039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09f0689f0714d8c8f45e24bd9adb533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.908141 30039 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.24.123.4:39087 (request call id 923) took 1101 ms. Trace:
I20240503 08:12:06.908342 30039 rpcz_store.cc:276] 0503 08:12:05.806971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:12:05.817489 (+ 10518us) service_pool.cc:225] Handling call
0503 08:12:05.887408 (+ 69919us) ts_tablet_manager.cc:903] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: Initiating tablet copy from peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
0503 08:12:06.900942 (+1013534us) tablet_copy_client.cc:353] Tablet copy session begun
0503 08:12:06.904646 (+ 3704us) tablet_metadata.cc:641] Metadata flushed
0503 08:12:06.904747 (+ 101us) tablet_copy_client.cc:482] Wrote new tablet metadata
0503 08:12:06.907826 (+ 3079us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 08:12:06.907968 (+ 142us) tablet_replica.cc:205] Creating consensus instance
0503 08:12:06.908085 (+ 117us) ts_tablet_manager.cc:958] Replica registered.
0503 08:12:06.908106 (+ 21us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":83,"tablet-copy.queue_time_us":69528,"thread_start_us":941291,"threads_started":5}
I20240503 08:12:06.909755 30039 tablet_copy_client.cc:783] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:06.910091 30039 tablet_copy_client.cc:647] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:06.992847 30039 tablet_copy_client.cc:523] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:06.996379 30039 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap starting.
W20240503 08:12:07.104208 27169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 111951us
W20240503 08:12:07.589087 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 48087.3us
I20240503 08:12:07.753304 30039 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap replayed 1/1 log segments. Stats: ops{read=166 overwritten=0 applied=166 ignored=0} inserts{seen=794 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:07.753909 30039 tablet_bootstrap.cc:492] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: Bootstrap complete.
I20240503 08:12:07.754310 30039 ts_tablet_manager.cc:1367] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: Time spent bootstrapping tablet: real 0.758s user 0.221s sys 0.013s
I20240503 08:12:07.756971 30039 raft_consensus.cc:348] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 160 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } }
I20240503 08:12:07.757383 30039 raft_consensus.cc:729] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Initialized, Role: LEARNER
I20240503 08:12:07.757678 30039 consensus_queue.cc:260] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 166, Last appended: 1.166, Last appended by leader: 166, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 160 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: true } }
I20240503 08:12:07.760061 30039 ts_tablet_manager.cc:1398] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:07.771973 27515 tablet_copy_service.cc:342] P 19fb518a6ede476f809c743054941e97: Request end of tablet copy session 36fc7a106457453ba91f05b78a5d342b-f09f0689f0714d8c8f45e24bd9adb533 received from {username='slave'} at 127.24.123.2:59799
I20240503 08:12:07.772128 27515 tablet_copy_service.cc:433] P 19fb518a6ede476f809c743054941e97: ending tablet copy session 36fc7a106457453ba91f05b78a5d342b-f09f0689f0714d8c8f45e24bd9adb533 on tablet f09f0689f0714d8c8f45e24bd9adb533 with peer 36fc7a106457453ba91f05b78a5d342b
I20240503 08:12:07.820488 27235 raft_consensus.cc:1203] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.159->[1.160-1.167] Dedup: 1.166->[1.167-1.167]
W20240503 08:12:07.829047 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 280162us
I20240503 08:12:08.004120 28981 raft_consensus.cc:1050] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97: attempting to promote NON_VOTER 36fc7a106457453ba91f05b78a5d342b to VOTER
I20240503 08:12:08.004947 28981 consensus_queue.cc:237] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 1.168, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } }
I20240503 08:12:08.008066 28749 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240503 08:12:08.009498 27625 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240503 08:12:08.010138 29931 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 08:12:08.014341 27624 raft_consensus.cc:1203] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.167->[1.168-1.169] Dedup: 1.168->[1.169-1.169]
I20240503 08:12:08.018851 27230 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEARNER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240503 08:12:08.019495 29811 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 08:12:08.023520 27752 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
W20240503 08:12:08.033502 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 79174.2us
I20240503 08:12:08.042759 29931 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 08:12:08.046608 29971 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
W20240503 08:12:08.138226 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 91694.3us
I20240503 08:12:08.147334 29811 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.169: config changed from index 160 to 169, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.175877 28749 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 160 to 169, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.265270 26961 catalog_manager.cc:5484] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 reported cstate change: config changed from index 160 to 169, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.285022 27100 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240503 08:12:08.286677 30075 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 08:12:08.289467 27104 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 160 to 169, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.348353 27230 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 160 to 169, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.371690 27750 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 160 to 169, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.374090 27490 consensus_queue.cc:237] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 1.170, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } }
I20240503 08:12:08.376924 28749 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 08:12:08.377581 29811 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 08:12:08.396479 27624 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 160 to 169, 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.426335 27235 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 08:12:08.427389 29811 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 08:12:08.440248 28981 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.171: config changed from index 169 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.440842 27235 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 169 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.448236 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f09f0689f0714d8c8f45e24bd9adb533 with cas_config_opid_index 169: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:08.449203 28751 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 169 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.450565 26954 catalog_manager.cc:5484] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 reported cstate change: config changed from index 169 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:08.457181 27084 tablet_service.cc:1514] Processing DeleteTablet for tablet f09f0689f0714d8c8f45e24bd9adb533 with delete_type TABLET_DATA_TOMBSTONED (TS f4d3aef6d5624cf4adc3859dbadccb80 not found in new config with opid_index 171) from {username='slave'} at 127.0.0.1:57128
I20240503 08:12:08.464332 27748 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 08:12:08.465224 30105 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 08:12:08.473860 27750 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 169 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
I20240503 08:12:08.481015 30119 tablet_replica.cc:330] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: stopping tablet replica
I20240503 08:12:08.481316 30119 raft_consensus.cc:2229] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:08.481730 30119 pending_rounds.cc:62] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: Trying to abort 1 pending ops.
I20240503 08:12:08.482164 30119 pending_rounds.cc:68] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: Aborting op as it isn't in flight: id { term: 1 index: 170 } timestamp: 7023509210018316288 op_type: WRITE_OP write_request { tablet_id: "f09f0689f0714d8c8f45e24bd9adb533" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\306O|]@\366\351\307\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""4c0076a6e4db668a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023509209067327488 authz_token { token_data: "\010\346\272\322\261\006\"3\n\005slave\022*\n e89fa94dfc184007962228c3c499e7e5\020\001\030\001 \001(\001" signature: "<redacted>""7\377\304p1\247\021\245\332\203<\264\254\300hM\233\017\004\303\302?\035\271pJ\303\231w\'\372v\177\373I\031\222T\350p eH\030\217\221\2416\375\262\231\010\273\277\303^\236PyZ\025\225\373Y" signing_key_seq_num: 0 } } request_id { client_id: "dc47dab7f58940c2ac7feeaed7e55991" seq_no: 532 first_incomplete_seq_no: 532 attempt_no: 0 }
I20240503 08:12:08.483132 30119 raft_consensus.cc:2258] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:08.484244 30119 ts_tablet_manager.cc:1875] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:08.494213 30119 ts_tablet_manager.cc:1888] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.170
I20240503 08:12:08.503031 30119 log.cc:1196] T f09f0689f0714d8c8f45e24bd9adb533 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/f09f0689f0714d8c8f45e24bd9adb533
I20240503 08:12:08.504120 26950 catalog_manager.cc:4830] TS f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): tablet f09f0689f0714d8c8f45e24bd9adb533 (table rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]) successfully deleted
I20240503 08:12:08.528293 27624 raft_consensus.cc:1261] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 08:12:08.536094 30075 consensus_queue.cc:1035] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 08:12:08.541571 27622 raft_consensus.cc:2932] T f09f0689f0714d8c8f45e24bd9adb533 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 169 to 171, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false } } }
W20240503 08:12:10.069728 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 44721us
W20240503 08:12:10.235810 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 57777.4us
W20240503 08:12:10.358191 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 53037us
W20240503 08:12:10.517714 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 45383.9us
I20240503 08:12:10.586020 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:10.590204 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:10.612402 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:10.620769 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:10.644201 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:12:10.764420 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 79169.5us
I20240503 08:12:10.871104 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:10.877044 27213 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:12:11.152350 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 212111us
W20240503 08:12:11.401253 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 85446.4us
W20240503 08:12:11.518347 27429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43185, user_credentials={real_user=slave}} blocked reactor thread for 44434.8us
W20240503 08:12:11.579538 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 87787.3us
W20240503 08:12:12.226789 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 84637.6us
W20240503 08:12:12.427228 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 96739.7us
W20240503 08:12:12.807704 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 47679.4us
W20240503 08:12:12.884641 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 77934.1us
I20240503 08:12:12.964224 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:12.977249 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:13.070628 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:13.075660 27213 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:13.148286 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:13.197618 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:13.269042 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:12:13.790246 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 62112.3us
W20240503 08:12:14.996719 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 49760.6us
W20240503 08:12:15.141346 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 52058.1us
W20240503 08:12:15.824997 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 46691.3us
W20240503 08:12:15.922946 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 141321us
W20240503 08:12:16.833756 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 122298us
W20240503 08:12:17.111426 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 59844.8us
W20240503 08:12:17.341866 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 48512.9us
W20240503 08:12:17.689478 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 345009us
W20240503 08:12:18.073997 27301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 85351.5us
I20240503 08:12:18.574726 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:18.591188 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 08:12:18.619880 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 71149.2us
W20240503 08:12:18.626928 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 63368.4us
I20240503 08:12:18.659267 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:18.671137 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:18.712240 27213 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:18.721153 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:12:18.756006 27558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 47508us
W20240503 08:12:18.765980 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 65662.5us
I20240503 08:12:19.081960 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:12:19.924925 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39261, user_credentials={real_user=slave}} blocked reactor thread for 56362.2us
W20240503 08:12:20.231128 27561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 84458.9us
W20240503 08:12:20.394863 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:43465, user_credentials={real_user=slave}} blocked reactor thread for 81039.9us
W20240503 08:12:20.666194 27037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 102238us
W20240503 08:12:20.737960 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 46855.7us
W20240503 08:12:21.109110 27431 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:41167, user_credentials={real_user=slave}} blocked reactor thread for 43943.9us
W20240503 08:12:21.527429 27428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:33503, user_credentials={real_user=slave}} blocked reactor thread for 65287.3us
W20240503 08:12:21.754283 27428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:43977, user_credentials={real_user=slave}} blocked reactor thread for 118309us
W20240503 08:12:21.839915 28751 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.092s sys 0.079s
W20240503 08:12:21.840049 28751 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.508s user 0.092s sys 0.079s
I20240503 08:12:22.828215 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:22.896325 27213 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:23.108667 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:23.139310 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:23.161195 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:23.189390 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:23.384929 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 08:12:23.999761 27038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:41395, user_credentials={real_user=slave}} blocked reactor thread for 55087us
I20240503 08:12:25.561331 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:25.564266 27213 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:25.567238 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:25.571204 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:25.574498 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:25.603008 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:25.631754 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
140b761e00e94e39b950d32863080556 | 127.24.123.62:40087 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.123.20:41695 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1980d01e2e3e4840a8ddfca4d4e366b2 | 127.24.123.7:41167 | HEALTHY | <none> | 0 | 0
19fb518a6ede476f809c743054941e97 | 127.24.123.4:41395 | HEALTHY | <none> | 3 | 0
2de33eb4315b4547889d1e6edae2f2dd | 127.24.123.5:43185 | HEALTHY | <none> | 4 | 0
36fc7a106457453ba91f05b78a5d342b | 127.24.123.2:33503 | HEALTHY | <none> | 3 | 0
d8d7c32cb9a241c99853a6639451c4ed | 127.24.123.6:43465 | HEALTHY | <none> | 0 | 0
f4d3aef6d5624cf4adc3859dbadccb80 | 127.24.123.1:39261 | HEALTHY | <none> | 3 | 0
fe8fdcdebbff4703a924f83646531115 | 127.24.123.3:43977 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.24.123.1 | experimental | 127.24.123.1:39261
local_ip_for_outbound_sockets | 127.24.123.2 | experimental | 127.24.123.2:33503
local_ip_for_outbound_sockets | 127.24.123.3 | experimental | 127.24.123.3:43977
local_ip_for_outbound_sockets | 127.24.123.4 | experimental | 127.24.123.4:41395
local_ip_for_outbound_sockets | 127.24.123.5 | experimental | 127.24.123.5:43185
local_ip_for_outbound_sockets | 127.24.123.6 | experimental | 127.24.123.6:43465
local_ip_for_outbound_sockets | 127.24.123.7 | experimental | 127.24.123.7:41167
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.123.1:39261
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.123.2:33503
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.123.3:43977
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.24.123.4:41395
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.24.123.5:43185
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.24.123.6:43465
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.24.123.7:41167
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.123.20:41695 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 11
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:12:25.687155 25068 log_verifier.cc:126] Checking tablet 03b4df9c9c0c4ec59b00338de1ae2920
I20240503 08:12:26.131811 25068 log_verifier.cc:177] Verified matching terms for 445 ops in tablet 03b4df9c9c0c4ec59b00338de1ae2920
I20240503 08:12:26.132911 25068 log_verifier.cc:126] Checking tablet 08b4767b5b4f4fb0bfc73c35a408a5ac
I20240503 08:12:26.551801 25068 log_verifier.cc:177] Verified matching terms for 419 ops in tablet 08b4767b5b4f4fb0bfc73c35a408a5ac
I20240503 08:12:26.552786 25068 log_verifier.cc:126] Checking tablet 43351c2d61024ae0afc96b6cf94d1741
I20240503 08:12:26.967350 25068 log_verifier.cc:177] Verified matching terms for 385 ops in tablet 43351c2d61024ae0afc96b6cf94d1741
I20240503 08:12:26.968236 25068 log_verifier.cc:126] Checking tablet 4c364e9ddad74e8e882df9080d9d5372
I20240503 08:12:27.409045 25068 log_verifier.cc:177] Verified matching terms for 471 ops in tablet 4c364e9ddad74e8e882df9080d9d5372
I20240503 08:12:27.410992 25068 log_verifier.cc:126] Checking tablet 89392b248b7746d5bf8b95ac9d662e43
I20240503 08:12:27.751575 25068 log_verifier.cc:177] Verified matching terms for 313 ops in tablet 89392b248b7746d5bf8b95ac9d662e43
I20240503 08:12:27.752452 25068 log_verifier.cc:126] Checking tablet 8e17a88a118b4ce591a266fd62b3d7a7
I20240503 08:12:28.233008 25068 log_verifier.cc:177] Verified matching terms for 522 ops in tablet 8e17a88a118b4ce591a266fd62b3d7a7
I20240503 08:12:28.234223 25068 log_verifier.cc:126] Checking tablet 8e4a8224c7c1430a9354e58bd16c7abd
I20240503 08:12:28.624282 25068 log_verifier.cc:177] Verified matching terms for 424 ops in tablet 8e4a8224c7c1430a9354e58bd16c7abd
I20240503 08:12:28.625270 25068 log_verifier.cc:126] Checking tablet 8f94c738c0f34d04bd933cc09c05d6a6
I20240503 08:12:29.053828 25068 log_verifier.cc:177] Verified matching terms for 442 ops in tablet 8f94c738c0f34d04bd933cc09c05d6a6
I20240503 08:12:29.054829 25068 log_verifier.cc:126] Checking tablet 921b4706d2bc4693ae5a0a10b310d5d4
I20240503 08:12:29.362848 25068 log_verifier.cc:177] Verified matching terms for 305 ops in tablet 921b4706d2bc4693ae5a0a10b310d5d4
I20240503 08:12:29.363644 25068 log_verifier.cc:126] Checking tablet a61402b1e1644ab7a0769b74e40e479b
I20240503 08:12:29.722818 25068 log_verifier.cc:177] Verified matching terms for 378 ops in tablet a61402b1e1644ab7a0769b74e40e479b
I20240503 08:12:29.723701 25068 log_verifier.cc:126] Checking tablet b8d5c3e2f8e6428aa457ba21d536cea5
I20240503 08:12:29.769263 25068 log_verifier.cc:177] Verified matching terms for 5 ops in tablet b8d5c3e2f8e6428aa457ba21d536cea5
I20240503 08:12:29.769381 25068 log_verifier.cc:126] Checking tablet c42929dcee3c461c8db91346ca1ab0f5
I20240503 08:12:30.223811 25068 log_verifier.cc:177] Verified matching terms for 469 ops in tablet c42929dcee3c461c8db91346ca1ab0f5
I20240503 08:12:30.224884 25068 log_verifier.cc:126] Checking tablet ccb3a8dde7ec4d318b0961c8bad8032c
I20240503 08:12:30.677882 25068 log_verifier.cc:177] Verified matching terms for 495 ops in tablet ccb3a8dde7ec4d318b0961c8bad8032c
I20240503 08:12:30.678987 25068 log_verifier.cc:126] Checking tablet dd41f1db8043464583458568a5900a7d
I20240503 08:12:31.148654 25068 log_verifier.cc:177] Verified matching terms for 466 ops in tablet dd41f1db8043464583458568a5900a7d
I20240503 08:12:31.149672 25068 log_verifier.cc:126] Checking tablet e8c93cddbbf84976baa817ab0d98a811
I20240503 08:12:31.485618 25068 log_verifier.cc:177] Verified matching terms for 349 ops in tablet e8c93cddbbf84976baa817ab0d98a811
I20240503 08:12:31.486471 25068 log_verifier.cc:126] Checking tablet f09f0689f0714d8c8f45e24bd9adb533
I20240503 08:12:31.787851 25068 log_verifier.cc:177] Verified matching terms for 279 ops in tablet f09f0689f0714d8c8f45e24bd9adb533
I20240503 08:12:31.795761 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.8:0
--local_ip_for_outbound_sockets=127.24.123.8
--webserver_interface=127.24.123.8
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--builtin_ntp_servers=127.24.123.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:12:32.148413 30678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:32.148670 30678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:32.148785 30678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:32.156251 30678 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:32.156385 30678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:32.156543 30678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.8
I20240503 08:12:32.165915 30678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.123.8
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.8
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:32.167171 30678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:32.168928 30678 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:32.181962 30685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:32.182503 30684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:32.182812 30687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:32.183302 30678 server_base.cc:1034] running on GCE node
I20240503 08:12:32.183923 30678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:32.185003 30678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:32.186280 30678 hybrid_clock.cc:648] HybridClock initialized: now 1714723952186181 us; error 86 us; skew 500 ppm
I20240503 08:12:32.186628 30678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:32.189126 30678 webserver.cc:460] Webserver started at http://127.24.123.8:41689/ using document root <none> and password file <none>
I20240503 08:12:32.189972 30678 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:32.190083 30678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:32.190400 30678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:32.193645 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "b4654488aec741f299e533fdf3715ce7"
format_stamp: "Formatted at 2024-05-03 08:12:32 on dist-test-slave-gkrw"
I20240503 08:12:32.194761 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "b4654488aec741f299e533fdf3715ce7"
format_stamp: "Formatted at 2024-05-03 08:12:32 on dist-test-slave-gkrw"
I20240503 08:12:32.200872 30678 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20240503 08:12:32.204288 30694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:32.205494 30678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:32.205646 30678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b4654488aec741f299e533fdf3715ce7"
format_stamp: "Formatted at 2024-05-03 08:12:32 on dist-test-slave-gkrw"
I20240503 08:12:32.205817 30678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:32.225183 30678 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:32.226032 30678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:32.226297 30678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:32.227236 30678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:32.229599 30678 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:32.229694 30678 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:32.229794 30678 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:32.229863 30678 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:32.264478 30678 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.8:35727
I20240503 08:12:32.264526 30806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.8:35727 every 8 connection(s)
I20240503 08:12:32.265828 30678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:12:32.275939 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 30678
I20240503 08:12:32.276145 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:12:32.278033 30807 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40087
I20240503 08:12:32.278411 30807 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:32.279429 30807 heartbeater.cc:511] Master 127.24.123.62:40087 requested a full tablet report, sending...
I20240503 08:12:32.280774 26961 ts_manager.cc:194] Registered new tserver with Master: b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727)
I20240503 08:12:32.281682 26961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.8:59323
I20240503 08:12:32.764619 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:32.773294 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:32.796510 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:32.820145 30742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:32.837572 27213 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:32.857141 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:32.860787 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:32.860841 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:33.063405 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:33.078008 27213 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:33.099943 27473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:33.100646 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:33.101569 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:33.121578 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:33.133585 30742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:33.156184 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:33.259729 27624 consensus_queue.cc:237] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 1.313, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:33.262948 27490 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 1 index: 314. (index mismatch)
I20240503 08:12:33.263648 30679 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
W20240503 08:12:33.271988 27559 consensus_peers.cc:485] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89392b248b7746d5bf8b95ac9d662e43. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:33.272019 27361 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 1 index: 314. (index mismatch)
I20240503 08:12:33.272019 27235 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 1 index: 314. (index mismatch)
I20240503 08:12:33.272800 30679 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20240503 08:12:33.273319 30679 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20240503 08:12:33.275053 27103 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 1 index: 314. (index mismatch)
I20240503 08:12:33.275732 30679 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20240503 08:12:33.282575 30679 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.314: config changed from index -1 to 314, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.283828 27234 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.314: config changed from index -1 to 314, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.286260 30807 heartbeater.cc:503] Master 127.24.123.62:40087 was elected leader, sending a full tablet report...
I20240503 08:12:33.293901 26954 catalog_manager.cc:5484] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: config changed from index -1 to 314, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.295940 27490 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.314: config changed from index -1 to 314, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.298331 27361 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.314: config changed from index -1 to 314, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.297204 27103 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.314: config changed from index -1 to 314, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.313688 27361 consensus_queue.cc:237] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 495, Committed index: 495, Last appended: 1.495, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:33.316829 27490 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 495. Preceding OpId from leader: term: 1 index: 496. (index mismatch)
I20240503 08:12:33.317716 27103 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 495. Preceding OpId from leader: term: 1 index: 496. (index mismatch)
I20240503 08:12:33.318320 28750 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 495. Preceding OpId from leader: term: 1 index: 496. (index mismatch)
I20240503 08:12:33.318918 27749 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 495. Preceding OpId from leader: term: 1 index: 496. (index mismatch)
I20240503 08:12:33.319617 30670 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240503 08:12:33.320204 30670 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240503 08:12:33.320621 30670 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240503 08:12:33.321069 30670 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
W20240503 08:12:33.323843 27299 consensus_peers.cc:485] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccb3a8dde7ec4d318b0961c8bad8032c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:33.327694 30670 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.496: config changed from index 216 to 496, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.332417 27103 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.496: config changed from index 216 to 496, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.332492 27749 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.496: config changed from index 216 to 496, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.342218 27490 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.496: config changed from index 216 to 496, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.343056 28750 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.496: config changed from index 216 to 496, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.347683 26961 catalog_manager.cc:5484] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 216 to 496, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:33.356015 27490 consensus_queue.cc:237] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 385, Committed index: 385, Last appended: 1.385, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:33.358593 27103 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240503 08:12:33.358816 27749 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240503 08:12:33.359308 30893 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240503 08:12:33.359980 27361 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240503 08:12:33.360525 30935 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240503 08:12:33.360853 27234 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240503 08:12:33.361166 30893 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240503 08:12:33.361409 30671 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240503 08:12:33.372650 30893 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.386: config changed from index -1 to 386, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
W20240503 08:12:33.373447 27429 consensus_peers.cc:485] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43351c2d61024ae0afc96b6cf94d1741. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:33.373934 27361 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index -1 to 386, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.376498 27103 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index -1 to 386, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.376585 27749 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index -1 to 386, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.380105 26955 catalog_manager.cc:5484] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index -1 to 386, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.383376 27234 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index -1 to 386, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.402725 27624 consensus_queue.cc:237] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 419, Committed index: 419, Last appended: 1.419, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 420 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:33.405308 27490 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 419. Preceding OpId from leader: term: 1 index: 420. (index mismatch)
I20240503 08:12:33.405730 28750 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 419. Preceding OpId from leader: term: 1 index: 420. (index mismatch)
W20240503 08:12:33.405864 27559 consensus_peers.cc:485] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08b4767b5b4f4fb0bfc73c35a408a5ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:33.405879 27361 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 419. Preceding OpId from leader: term: 1 index: 420. (index mismatch)
I20240503 08:12:33.406072 30900 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 420, Last known committed idx: 419, Time since last communication: 0.000s
I20240503 08:12:33.406316 30908 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 420, Last known committed idx: 419, Time since last communication: 0.000s
I20240503 08:12:33.406756 30900 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 420, Last known committed idx: 419, Time since last communication: 0.000s
I20240503 08:12:33.407656 27749 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 419. Preceding OpId from leader: term: 1 index: 420. (index mismatch)
I20240503 08:12:33.408231 30900 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 420, Last known committed idx: 419, Time since last communication: 0.000s
I20240503 08:12:33.421766 30907 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.420: config changed from index 116 to 420, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 420 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.422398 27361 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.420: config changed from index 116 to 420, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 420 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.428095 26960 catalog_manager.cc:5484] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 116 to 420, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 420 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.429526 27749 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.420: config changed from index 116 to 420, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 420 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.440009 28750 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.420: config changed from index 116 to 420, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 420 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.439033 27490 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.420: config changed from index 116 to 420, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 420 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.459769 27234 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:33.462431 28750 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 08:12:33.462481 27490 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 08:12:33.463016 30825 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:33.463135 27361 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 08:12:33.463519 27624 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 08:12:33.464149 30825 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:33.464630 30825 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:33.466389 30825 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:12:33.471329 27169 consensus_peers.cc:485] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8d5c3e2f8e6428aa457ba21d536cea5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:33.478753 30825 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.479475 27490 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.481302 27624 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.483671 27361 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.484290 28746 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.488373 26959 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 5 to 6, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 3 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:33.844173 31015 ts_tablet_manager.cc:902] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:12:33.845854 31015 tablet_copy_client.cc:308] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43185
I20240503 08:12:33.851855 27645 tablet_copy_service.cc:140] P 2de33eb4315b4547889d1e6edae2f2dd: Received BeginTabletCopySession request for tablet 89392b248b7746d5bf8b95ac9d662e43 from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:55327)
I20240503 08:12:33.851979 27645 tablet_copy_service.cc:161] P 2de33eb4315b4547889d1e6edae2f2dd: Beginning new tablet copy session on tablet 89392b248b7746d5bf8b95ac9d662e43 from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:55327: session id = b4654488aec741f299e533fdf3715ce7-89392b248b7746d5bf8b95ac9d662e43
I20240503 08:12:33.853776 27645 tablet_copy_source_session.cc:205] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:33.857676 31015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89392b248b7746d5bf8b95ac9d662e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.866302 31015 tablet_copy_client.cc:783] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:33.866597 31015 tablet_copy_client.cc:647] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:33.869208 31015 tablet_copy_client.cc:523] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:33.872100 31015 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:33.904811 31021 ts_tablet_manager.cc:902] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:12:33.905537 31022 ts_tablet_manager.cc:902] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:12:33.910461 31021 tablet_copy_client.cc:308] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:43977
I20240503 08:12:33.918242 27385 tablet_copy_service.cc:140] P fe8fdcdebbff4703a924f83646531115: Received BeginTabletCopySession request for tablet ccb3a8dde7ec4d318b0961c8bad8032c from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:36441)
I20240503 08:12:33.918407 27385 tablet_copy_service.cc:161] P fe8fdcdebbff4703a924f83646531115: Beginning new tablet copy session on tablet ccb3a8dde7ec4d318b0961c8bad8032c from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:36441: session id = b4654488aec741f299e533fdf3715ce7-ccb3a8dde7ec4d318b0961c8bad8032c
I20240503 08:12:33.920816 31015 log.cc:824] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:33.921026 27385 tablet_copy_source_session.cc:205] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:33.922508 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb3a8dde7ec4d318b0961c8bad8032c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.923198 31022 tablet_copy_client.cc:308] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:41395
I20240503 08:12:33.929272 31021 tablet_copy_client.cc:783] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:33.929592 31021 tablet_copy_client.cc:647] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:33.933108 31021 tablet_copy_client.cc:523] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:33.933740 27515 tablet_copy_service.cc:140] P 19fb518a6ede476f809c743054941e97: Received BeginTabletCopySession request for tablet 43351c2d61024ae0afc96b6cf94d1741 from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:51009)
I20240503 08:12:33.933887 27515 tablet_copy_service.cc:161] P 19fb518a6ede476f809c743054941e97: Beginning new tablet copy session on tablet 43351c2d61024ae0afc96b6cf94d1741 from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:51009: session id = b4654488aec741f299e533fdf3715ce7-43351c2d61024ae0afc96b6cf94d1741
I20240503 08:12:33.936983 27515 tablet_copy_source_session.cc:205] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:33.938392 31022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43351c2d61024ae0afc96b6cf94d1741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.950635 31021 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:33.957494 31022 tablet_copy_client.cc:783] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:33.957966 31022 tablet_copy_client.cc:647] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:33.961267 31022 tablet_copy_client.cc:523] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:33.970463 31022 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:33.972720 31032 ts_tablet_manager.cc:902] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185)
I20240503 08:12:34.007870 31032 tablet_copy_client.cc:308] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43185
I20240503 08:12:34.008574 27645 tablet_copy_service.cc:140] P 2de33eb4315b4547889d1e6edae2f2dd: Received BeginTabletCopySession request for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:55327)
I20240503 08:12:34.008709 27645 tablet_copy_service.cc:161] P 2de33eb4315b4547889d1e6edae2f2dd: Beginning new tablet copy session on tablet 08b4767b5b4f4fb0bfc73c35a408a5ac from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:55327: session id = b4654488aec741f299e533fdf3715ce7-08b4767b5b4f4fb0bfc73c35a408a5ac
I20240503 08:12:34.011518 27645 tablet_copy_source_session.cc:205] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:34.013008 31032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.027621 31032 tablet_copy_client.cc:783] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:34.027963 31032 tablet_copy_client.cc:647] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:34.031383 31032 tablet_copy_client.cc:523] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:34.036821 31032 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:34.111107 31048 ts_tablet_manager.cc:902] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503)
I20240503 08:12:34.146521 31048 tablet_copy_client.cc:308] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:33503
I20240503 08:12:34.153908 27255 tablet_copy_service.cc:140] P 36fc7a106457453ba91f05b78a5d342b: Received BeginTabletCopySession request for tablet b8d5c3e2f8e6428aa457ba21d536cea5 from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:60703)
I20240503 08:12:34.154083 27255 tablet_copy_service.cc:161] P 36fc7a106457453ba91f05b78a5d342b: Beginning new tablet copy session on tablet b8d5c3e2f8e6428aa457ba21d536cea5 from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:60703: session id = b4654488aec741f299e533fdf3715ce7-b8d5c3e2f8e6428aa457ba21d536cea5
I20240503 08:12:34.156800 27255 tablet_copy_source_session.cc:205] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:34.159117 31048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d5c3e2f8e6428aa457ba21d536cea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.167938 31048 tablet_copy_client.cc:783] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:34.168255 31048 tablet_copy_client.cc:647] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:34.170097 31048 tablet_copy_client.cc:523] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:34.173803 31048 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:34.221081 31048 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:34.224125 31048 tablet_bootstrap.cc:492] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:34.224776 31048 ts_tablet_manager.cc:1367] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.005s
I20240503 08:12:34.240713 31048 raft_consensus.cc:348] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.246028 31048 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:34.246776 31048 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.259663 31048 ts_tablet_manager.cc:1398] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.033s user 0.009s sys 0.004s
I20240503 08:12:34.268157 27255 tablet_copy_service.cc:342] P 36fc7a106457453ba91f05b78a5d342b: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-b8d5c3e2f8e6428aa457ba21d536cea5 received from {username='slave'} at 127.24.123.8:60703
I20240503 08:12:34.268304 27255 tablet_copy_service.cc:433] P 36fc7a106457453ba91f05b78a5d342b: ending tablet copy session b4654488aec741f299e533fdf3715ce7-b8d5c3e2f8e6428aa457ba21d536cea5 on tablet b8d5c3e2f8e6428aa457ba21d536cea5 with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:34.543690 31015 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=314 overwritten=0 applied=314 ignored=0} inserts{seen=986 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:34.544293 31015 tablet_bootstrap.cc:492] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:34.544719 31015 ts_tablet_manager.cc:1367] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.673s user 0.346s sys 0.039s
I20240503 08:12:34.546535 31015 raft_consensus.cc:348] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.546959 31015 raft_consensus.cc:729] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:34.547312 31015 consensus_queue.cc:260] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 314, Last appended: 1.314, Last appended by leader: 314, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.547950 31015 ts_tablet_manager.cc:1398] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:34.548956 27645 tablet_copy_service.cc:342] P 2de33eb4315b4547889d1e6edae2f2dd: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-89392b248b7746d5bf8b95ac9d662e43 received from {username='slave'} at 127.24.123.8:55327
I20240503 08:12:34.549085 27645 tablet_copy_service.cc:433] P 2de33eb4315b4547889d1e6edae2f2dd: ending tablet copy session b4654488aec741f299e533fdf3715ce7-89392b248b7746d5bf8b95ac9d662e43 on tablet 89392b248b7746d5bf8b95ac9d662e43 with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:34.603736 30762 raft_consensus.cc:1203] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[]
I20240503 08:12:34.680636 31032 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=420 overwritten=0 applied=420 ignored=0} inserts{seen=1048 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:34.681388 31032 tablet_bootstrap.cc:492] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:34.681793 31032 ts_tablet_manager.cc:1367] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.645s user 0.449s sys 0.067s
I20240503 08:12:34.682839 31032 raft_consensus.cc:348] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 420 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.683379 31032 raft_consensus.cc:729] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:34.683666 31032 consensus_queue.cc:260] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 420, Last appended: 1.420, Last appended by leader: 420, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 420 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.685366 31022 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=386 overwritten=0 applied=386 ignored=0} inserts{seen=1071 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:34.685786 31022 tablet_bootstrap.cc:492] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:34.686183 31022 ts_tablet_manager.cc:1367] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.717s user 0.403s sys 0.071s
I20240503 08:12:34.687161 31022 raft_consensus.cc:348] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.687428 31032 ts_tablet_manager.cc:1398] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:34.687533 31022 raft_consensus.cc:729] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:34.687780 31022 consensus_queue.cc:260] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 386, Last appended: 1.386, Last appended by leader: 386, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.688482 31022 ts_tablet_manager.cc:1398] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:34.689996 27645 tablet_copy_service.cc:342] P 2de33eb4315b4547889d1e6edae2f2dd: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-08b4767b5b4f4fb0bfc73c35a408a5ac received from {username='slave'} at 127.24.123.8:55327
I20240503 08:12:34.690129 27645 tablet_copy_service.cc:433] P 2de33eb4315b4547889d1e6edae2f2dd: ending tablet copy session b4654488aec741f299e533fdf3715ce7-08b4767b5b4f4fb0bfc73c35a408a5ac on tablet 08b4767b5b4f4fb0bfc73c35a408a5ac with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:34.695873 27515 tablet_copy_service.cc:342] P 19fb518a6ede476f809c743054941e97: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-43351c2d61024ae0afc96b6cf94d1741 received from {username='slave'} at 127.24.123.8:51009
I20240503 08:12:34.695986 27515 tablet_copy_service.cc:433] P 19fb518a6ede476f809c743054941e97: ending tablet copy session b4654488aec741f299e533fdf3715ce7-43351c2d61024ae0afc96b6cf94d1741 on tablet 43351c2d61024ae0afc96b6cf94d1741 with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:34.773123 30762 raft_consensus.cc:1203] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.419->[1.420-1.420] Dedup: 1.420->[]
I20240503 08:12:34.784164 31021 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=496 overwritten=0 applied=496 ignored=0} inserts{seen=1178 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:34.784500 31021 tablet_bootstrap.cc:492] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:34.784740 31021 ts_tablet_manager.cc:1367] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.834s user 0.533s sys 0.055s
I20240503 08:12:34.785333 31021 raft_consensus.cc:348] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.785569 31021 raft_consensus.cc:729] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:34.785714 31021 consensus_queue.cc:260] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 496, Last appended: 1.496, Last appended by leader: 496, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:34.786212 31021 ts_tablet_manager.cc:1398] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:34.787043 27385 tablet_copy_service.cc:342] P fe8fdcdebbff4703a924f83646531115: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-ccb3a8dde7ec4d318b0961c8bad8032c received from {username='slave'} at 127.24.123.8:36441
I20240503 08:12:34.787148 27385 tablet_copy_service.cc:433] P fe8fdcdebbff4703a924f83646531115: ending tablet copy session b4654488aec741f299e533fdf3715ce7-ccb3a8dde7ec4d318b0961c8bad8032c on tablet ccb3a8dde7ec4d318b0961c8bad8032c with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:34.839617 30762 raft_consensus.cc:1203] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.385->[1.386-1.386] Dedup: 1.386->[]
I20240503 08:12:35.002308 30762 raft_consensus.cc:1203] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.495->[1.496-1.496] Dedup: 1.496->[]
I20240503 08:12:35.065114 31089 raft_consensus.cc:1050] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:35.065845 31089 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 3, Current term: 3, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.068307 27624 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 08:12:35.068295 27490 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 08:12:35.068365 28750 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 08:12:35.068763 30762 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 3 LEARNER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 08:12:35.069108 31135 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:35.069589 30964 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:35.069882 30762 raft_consensus.cc:1203] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.313->[1.314-1.314] Dedup: 1.314->[]
I20240503 08:12:35.070066 31136 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:35.070590 31087 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:35.072470 27361 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 3 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 08:12:35.073185 31089 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:35.084547 31089 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.085111 27361 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.085891 27624 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.086213 27490 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.089089 28750 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.090322 30762 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.091053 26955 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: config changed from index 6 to 7, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:35.208755 31066 raft_consensus.cc:1050] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:35.209825 31066 consensus_queue.cc:237] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 420, Committed index: 420, Last appended: 1.420, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 421 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.213331 27361 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 420. Preceding OpId from leader: term: 1 index: 421. (index mismatch)
I20240503 08:12:35.213433 28750 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 420. Preceding OpId from leader: term: 1 index: 421. (index mismatch)
I20240503 08:12:35.213593 30762 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 420. Preceding OpId from leader: term: 1 index: 421. (index mismatch)
I20240503 08:12:35.214017 31153 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 421, Last known committed idx: 420, Time since last communication: 0.000s
I20240503 08:12:35.214322 30907 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 421, Last known committed idx: 420, Time since last communication: 0.000s
I20240503 08:12:35.214454 27749 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 420. Preceding OpId from leader: term: 1 index: 421. (index mismatch)
I20240503 08:12:35.214854 31067 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 421, Last known committed idx: 420, Time since last communication: 0.000s
I20240503 08:12:35.215166 30907 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 421, Last known committed idx: 420, Time since last communication: 0.000s
I20240503 08:12:35.215234 27490 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 420. Preceding OpId from leader: term: 1 index: 421. (index mismatch)
I20240503 08:12:35.215837 31153 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 421, Last known committed idx: 420, Time since last communication: 0.000s
I20240503 08:12:35.231679 27749 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.421: config changed from index 420 to 421, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 421 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.231998 27361 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.421: config changed from index 420 to 421, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 421 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.232595 27490 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.421: config changed from index 420 to 421, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 421 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.230712 31153 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.421: config changed from index 420 to 421, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 421 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.240824 26958 catalog_manager.cc:5484] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 420 to 421, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 421 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.240958 30762 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.421: config changed from index 420 to 421, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 421 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.240922 28750 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.421: config changed from index 420 to 421, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 421 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.260819 27624 consensus_queue.cc:237] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 421, Committed index: 421, Last appended: 1.421, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 422 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.264039 28750 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 421. Preceding OpId from leader: term: 1 index: 422. (index mismatch)
I20240503 08:12:35.264552 30762 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 421. Preceding OpId from leader: term: 1 index: 422. (index mismatch)
I20240503 08:12:35.264974 27490 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 421. Preceding OpId from leader: term: 1 index: 422. (index mismatch)
I20240503 08:12:35.267000 31162 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 422, Last known committed idx: 421, Time since last communication: 0.000s
I20240503 08:12:35.267482 31162 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 422, Last known committed idx: 421, Time since last communication: 0.000s
I20240503 08:12:35.267930 31162 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 422, Last known committed idx: 421, Time since last communication: 0.000s
I20240503 08:12:35.268913 27749 raft_consensus.cc:1261] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 421. Preceding OpId from leader: term: 1 index: 422. (index mismatch)
I20240503 08:12:35.270081 31162 consensus_queue.cc:1035] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 422, Last known committed idx: 421, Time since last communication: 0.000s
I20240503 08:12:35.270330 27234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5"
dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35022
I20240503 08:12:35.270555 27234 raft_consensus.cc:595] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 LEADER]: Received request to transfer leadership
I20240503 08:12:35.271876 31153 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.422: config changed from index 421 to 422, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 422 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.272939 27749 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.422: config changed from index 421 to 422, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 422 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.276157 31077 raft_consensus.cc:1050] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:35.277072 31077 consensus_queue.cc:237] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 1.386, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.277678 27490 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.422: config changed from index 421 to 422, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 422 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.277983 26958 catalog_manager.cc:5484] T 08b4767b5b4f4fb0bfc73c35a408a5ac P d8d7c32cb9a241c99853a6639451c4ed reported cstate change: config changed from index 421 to 422, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 422 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.279016 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac with cas_config_opid_index 421: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:35.279114 30762 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.422: config changed from index 421 to 422, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 422 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.280206 28750 raft_consensus.cc:2932] T 08b4767b5b4f4fb0bfc73c35a408a5ac P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.422: config changed from index 421 to 422, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 422 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.289194 27749 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240503 08:12:35.289631 31078 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240503 08:12:35.290038 27344 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b4767b5b4f4fb0bfc73c35a408a5ac with delete_type TABLET_DATA_TOMBSTONED (TS fe8fdcdebbff4703a924f83646531115 not found in new config with opid_index 422) from {username='slave'} at 127.0.0.1:60962
I20240503 08:12:35.294013 27103 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240503 08:12:35.294947 31168 tablet_replica.cc:330] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115: stopping tablet replica
I20240503 08:12:35.295032 27361 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240503 08:12:35.295244 31168 raft_consensus.cc:2229] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:35.295508 31168 raft_consensus.cc:2258] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.295511 27234 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240503 08:12:35.296042 31076 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240503 08:12:35.297024 31077 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240503 08:12:35.297283 31078 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240503 08:12:35.297562 30762 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240503 08:12:35.298038 31076 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240503 08:12:35.303450 31168 ts_tablet_manager.cc:1875] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:35.314021 31076 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.387: config changed from index 386 to 387, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.317414 30762 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 386 to 387, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.317590 27234 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 386 to 387, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.318529 27103 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 386 to 387, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.319526 27361 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 386 to 387, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.321434 31168 ts_tablet_manager.cc:1888] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.421
I20240503 08:12:35.321576 31168 log.cc:1196] T 08b4767b5b4f4fb0bfc73c35a408a5ac P fe8fdcdebbff4703a924f83646531115: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/08b4767b5b4f4fb0bfc73c35a408a5ac
I20240503 08:12:35.322693 26947 catalog_manager.cc:4830] TS fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): tablet 08b4767b5b4f4fb0bfc73c35a408a5ac (table rebalance_test_table_1 [id=7a8676c1afa444b88243ef4c576fb868]) successfully deleted
I20240503 08:12:35.323310 26958 catalog_manager.cc:5484] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 reported cstate change: config changed from index 386 to 387, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.326467 27749 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 386 to 387, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.331696 27490 consensus_queue.cc:237] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 387, Committed index: 387, Last appended: 1.387, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.333966 27361 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20240503 08:12:35.334071 30762 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20240503 08:12:35.334599 27234 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20240503 08:12:35.334542 31078 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240503 08:12:35.335089 31078 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240503 08:12:35.335346 31077 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240503 08:12:35.337132 27749 raft_consensus.cc:1261] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20240503 08:12:35.337564 31078 consensus_queue.cc:1035] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240503 08:12:35.338436 31078 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.388: config changed from index 387 to 388, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.339943 30762 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.340425 27749 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.339624 27234 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.344811 26959 catalog_manager.cc:5484] T 43351c2d61024ae0afc96b6cf94d1741 P b4654488aec741f299e533fdf3715ce7 reported cstate change: config changed from index 387 to 388, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.348413 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43351c2d61024ae0afc96b6cf94d1741 with cas_config_opid_index 387: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:35.349225 27361 raft_consensus.cc:2932] T 43351c2d61024ae0afc96b6cf94d1741 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, VOTER f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.354730 27084 tablet_service.cc:1514] Processing DeleteTablet for tablet 43351c2d61024ae0afc96b6cf94d1741 with delete_type TABLET_DATA_TOMBSTONED (TS f4d3aef6d5624cf4adc3859dbadccb80 not found in new config with opid_index 388) from {username='slave'} at 127.0.0.1:57128
I20240503 08:12:35.355562 31184 tablet_replica.cc:330] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80: stopping tablet replica
I20240503 08:12:35.355732 31184 raft_consensus.cc:2229] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:35.355958 31184 raft_consensus.cc:2258] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.357692 31184 ts_tablet_manager.cc:1875] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:35.364085 31184 ts_tablet_manager.cc:1888] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.387
I20240503 08:12:35.364209 31184 log.cc:1196] T 43351c2d61024ae0afc96b6cf94d1741 P f4d3aef6d5624cf4adc3859dbadccb80: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/43351c2d61024ae0afc96b6cf94d1741
I20240503 08:12:35.364989 26950 catalog_manager.cc:4830] TS f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): tablet 43351c2d61024ae0afc96b6cf94d1741 (table rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]) successfully deleted
I20240503 08:12:35.388790 27103 consensus_queue.cc:237] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.349, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:35.391592 28750 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240503 08:12:35.392010 27624 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240503 08:12:35.392581 31202 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240503 08:12:35.392966 31034 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
W20240503 08:12:35.401075 27038 consensus_peers.cc:485] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8c93cddbbf84976baa817ab0d98a811. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:35.403831 27490 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240503 08:12:35.403858 27234 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240503 08:12:35.404431 31202 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240503 08:12:35.405187 31203 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240503 08:12:35.419759 31202 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.350: config changed from index 76 to 350, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.423385 31133 raft_consensus.cc:1050] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:35.423921 27623 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 76 to 350, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.424342 31133 consensus_queue.cc:237] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 496, Committed index: 496, Last appended: 1.496, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.425663 27234 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 76 to 350, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.431326 26959 catalog_manager.cc:5484] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: config changed from index 76 to 350, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:35.432230 27491 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 76 to 350, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.435259 27103 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 496. Preceding OpId from leader: term: 1 index: 497. (index mismatch)
I20240503 08:12:35.435413 27749 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 496. Preceding OpId from leader: term: 1 index: 497. (index mismatch)
I20240503 08:12:35.435940 30762 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 496. Preceding OpId from leader: term: 1 index: 497. (index mismatch)
I20240503 08:12:35.435858 31080 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 08:12:35.436183 30921 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 08:12:35.436482 27494 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 496. Preceding OpId from leader: term: 1 index: 497. (index mismatch)
I20240503 08:12:35.437433 30921 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 08:12:35.437944 30921 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 08:12:35.443451 28750 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 496. Preceding OpId from leader: term: 1 index: 497. (index mismatch)
I20240503 08:12:35.443966 31080 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 08:12:35.448071 28746 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 76 to 350, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.448520 31080 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.497: config changed from index 496 to 497, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.449004 30762 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.497: config changed from index 496 to 497, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.455922 27103 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.497: config changed from index 496 to 497, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.459532 27749 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.497: config changed from index 496 to 497, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.462448 26954 catalog_manager.cc:5484] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7 reported cstate change: config changed from index 496 to 497, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.474575 28750 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.497: config changed from index 496 to 497, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.475404 27490 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.497: config changed from index 496 to 497, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.483867 27361 consensus_queue.cc:237] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 497, Committed index: 497, Last appended: 1.497, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.485991 30762 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 497. Preceding OpId from leader: term: 1 index: 498. (index mismatch)
I20240503 08:12:35.486445 27103 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 497. Preceding OpId from leader: term: 1 index: 498. (index mismatch)
I20240503 08:12:35.486548 31080 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240503 08:12:35.487023 31222 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240503 08:12:35.487411 28750 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 497. Preceding OpId from leader: term: 1 index: 498. (index mismatch)
I20240503 08:12:35.488147 31133 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240503 08:12:35.488210 27490 raft_consensus.cc:1261] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 497. Preceding OpId from leader: term: 1 index: 498. (index mismatch)
I20240503 08:12:35.489527 31080 consensus_queue.cc:1035] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240503 08:12:35.490895 31133 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.498: config changed from index 497 to 498, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.491552 27490 consensus_queue.cc:237] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 445, Committed index: 445, Last appended: 1.445, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:35.492446 27491 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.498: config changed from index 497 to 498, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.494369 28746 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 445. Preceding OpId from leader: term: 1 index: 446. (index mismatch)
I20240503 08:12:35.494894 27623 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 445. Preceding OpId from leader: term: 1 index: 446. (index mismatch)
I20240503 08:12:35.495249 31076 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 446, Last known committed idx: 445, Time since last communication: 0.000s
I20240503 08:12:35.496136 31178 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 446, Last known committed idx: 445, Time since last communication: 0.000s
I20240503 08:12:35.496659 26947 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccb3a8dde7ec4d318b0961c8bad8032c with cas_config_opid_index 497: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:35.497062 28746 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.498: config changed from index 497 to 498, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.497684 26959 catalog_manager.cc:5484] T ccb3a8dde7ec4d318b0961c8bad8032c P 19fb518a6ede476f809c743054941e97 reported cstate change: config changed from index 497 to 498, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.500891 30762 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.498: config changed from index 497 to 498, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.501608 27097 raft_consensus.cc:2932] T ccb3a8dde7ec4d318b0961c8bad8032c P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.498: config changed from index 497 to 498, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.502705 31136 raft_consensus.cc:979] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: : Instructing follower 19fb518a6ede476f809c743054941e97 to start an election
I20240503 08:12:35.502830 31136 raft_consensus.cc:1067] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 LEADER]: Signalling peer 19fb518a6ede476f809c743054941e97 to start an election
I20240503 08:12:35.503669 27494 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5"
dest_uuid: "19fb518a6ede476f809c743054941e97"
from {username='slave'} at 127.24.123.2:59799
I20240503 08:12:35.503888 27494 raft_consensus.cc:482] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:35.503996 27494 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:35.508108 27234 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 445. Preceding OpId from leader: term: 1 index: 446. (index mismatch)
I20240503 08:12:35.508740 31076 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 446, Last known committed idx: 445, Time since last communication: 0.000s
I20240503 08:12:35.509224 31077 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.446: config changed from index 171 to 446, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.509851 27624 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.446: config changed from index 171 to 446, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
W20240503 08:12:35.510613 27429 consensus_peers.cc:485] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03b4df9c9c0c4ec59b00338de1ae2920. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:35.511615 27749 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 445. Preceding OpId from leader: term: 1 index: 446. (index mismatch)
I20240503 08:12:35.512463 31076 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 446, Last known committed idx: 445, Time since last communication: 0.000s
I20240503 08:12:35.514106 27749 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.446: config changed from index 171 to 446, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.515990 28746 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.446: config changed from index 171 to 446, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.516438 27494 raft_consensus.cc:504] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.517474 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:12:35.517657 27361 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:35.518007 26958 catalog_manager.cc:5484] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index 171 to 446, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.520123 27361 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 4.
I20240503 08:12:35.520628 27494 leader_election.cc:288] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 4 election: Requested vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727)
I20240503 08:12:35.521054 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "36fc7a106457453ba91f05b78a5d342b"
I20240503 08:12:35.521248 27235 raft_consensus.cc:3032] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:12:35.521342 27235 raft_consensus.cc:729] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 36fc7a106457453ba91f05b78a5d342b, State: Running, Role: LEADER
I20240503 08:12:35.521513 30762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "b4654488aec741f299e533fdf3715ce7"
I20240503 08:12:35.521795 30762 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:35.521670 27235 consensus_queue.cc:260] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.522209 27235 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:35.522913 27733 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb3a8dde7ec4d318b0961c8bad8032c with delete_type TABLET_DATA_TOMBSTONED (TS d8d7c32cb9a241c99853a6639451c4ed not found in new config with opid_index 498) from {username='slave'} at 127.0.0.1:42730
I20240503 08:12:35.524565 30762 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 4.
I20240503 08:12:35.524885 27235 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 4.
I20240503 08:12:35.527262 27428 leader_election.cc:302] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, 36fc7a106457453ba91f05b78a5d342b, b4654488aec741f299e533fdf3715ce7, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:12:35.527282 28746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2"
I20240503 08:12:35.527551 31077 raft_consensus.cc:2781] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:12:35.527932 31077 raft_consensus.cc:686] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 4 LEADER]: Becoming Leader. State: Replica: 19fb518a6ede476f809c743054941e97, State: Running, Role: LEADER
I20240503 08:12:35.527947 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d5c3e2f8e6428aa457ba21d536cea5" candidate_uuid: "19fb518a6ede476f809c743054941e97" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd"
I20240503 08:12:35.528162 27624 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:35.528322 31077 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 4, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.530167 28746 raft_consensus.cc:3037] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:35.530200 27624 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 4.
I20240503 08:12:35.532955 28746 raft_consensus.cc:2454] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb518a6ede476f809c743054941e97 in term 4.
I20240503 08:12:35.535768 31066 raft_consensus.cc:1050] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:35.536926 31066 consensus_queue.cc:237] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 1.314, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.539479 27491 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 314. Preceding OpId from leader: term: 1 index: 315. (index mismatch)
I20240503 08:12:35.540350 31067 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20240503 08:12:35.540326 26959 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 reported cstate change: term changed from 3 to 4, leader changed from 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) to 19fb518a6ede476f809c743054941e97 (127.24.123.4). New cstate: current_term: 4 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:35.540771 27361 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 314. Preceding OpId from leader: term: 1 index: 315. (index mismatch)
I20240503 08:12:35.541046 27097 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 314. Preceding OpId from leader: term: 1 index: 315. (index mismatch)
I20240503 08:12:35.541303 31228 tablet_replica.cc:330] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed: stopping tablet replica
I20240503 08:12:35.541492 31228 raft_consensus.cc:2229] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:35.541860 31228 raft_consensus.cc:2258] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.542271 31067 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20240503 08:12:35.542732 31067 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20240503 08:12:35.543134 30762 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 314. Preceding OpId from leader: term: 1 index: 315. (index mismatch)
I20240503 08:12:35.543754 31067 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20240503 08:12:35.544564 31228 ts_tablet_manager.cc:1875] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:35.550940 27235 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 314. Preceding OpId from leader: term: 1 index: 315. (index mismatch)
I20240503 08:12:35.551508 30907 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20240503 08:12:35.552160 31228 ts_tablet_manager.cc:1888] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.497
I20240503 08:12:35.552316 31228 log.cc:1196] T ccb3a8dde7ec4d318b0961c8bad8032c P d8d7c32cb9a241c99853a6639451c4ed: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/ccb3a8dde7ec4d318b0961c8bad8032c
I20240503 08:12:35.553210 26950 catalog_manager.cc:4830] TS d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465): tablet ccb3a8dde7ec4d318b0961c8bad8032c (table rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]) successfully deleted
I20240503 08:12:35.559689 27234 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.446: config changed from index 171 to 446, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.565254 31153 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.315: config changed from index 314 to 315, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.565744 27363 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.315: config changed from index 314 to 315, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.568115 30761 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.315: config changed from index 314 to 315, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.569769 27097 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.315: config changed from index 314 to 315, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.571185 27491 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.315: config changed from index 314 to 315, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.574146 26959 catalog_manager.cc:5484] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index 314 to 315, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:35.576992 27234 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.315: config changed from index 314 to 315, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.580687 27624 consensus_queue.cc:237] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 1.315, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:35.583092 30761 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240503 08:12:35.583382 27097 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240503 08:12:35.583457 27361 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240503 08:12:35.583629 31162 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240503 08:12:35.584254 27234 raft_consensus.cc:1261] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 2de33eb4315b4547889d1e6edae2f2dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240503 08:12:35.584203 31162 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240503 08:12:35.585361 31066 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240503 08:12:35.585846 31066 consensus_queue.cc:1035] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20240503 08:12:35.587827 31162 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 LEADER]: Committing config change with OpId 1.316: config changed from index 315 to 316, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.588477 27097 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 315 to 316, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.592054 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 89392b248b7746d5bf8b95ac9d662e43 with cas_config_opid_index 315: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:35.591725 27361 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 315 to 316, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.594537 30761 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 315 to 316, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.595062 26954 catalog_manager.cc:5484] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index 315 to 316, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "2de33eb4315b4547889d1e6edae2f2dd" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:35.599006 27230 raft_consensus.cc:2932] T 89392b248b7746d5bf8b95ac9d662e43 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 315 to 316, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:35.599568 27473 tablet_service.cc:1514] Processing DeleteTablet for tablet 89392b248b7746d5bf8b95ac9d662e43 with delete_type TABLET_DATA_TOMBSTONED (TS 19fb518a6ede476f809c743054941e97 not found in new config with opid_index 316) from {username='slave'} at 127.0.0.1:56480
I20240503 08:12:35.600347 31240 tablet_replica.cc:330] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97: stopping tablet replica
I20240503 08:12:35.600534 31240 raft_consensus.cc:2229] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:35.600754 31240 raft_consensus.cc:2258] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.603322 31240 ts_tablet_manager.cc:1875] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:35.610138 31240 ts_tablet_manager.cc:1888] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.315
I20240503 08:12:35.610255 31240 log.cc:1196] T 89392b248b7746d5bf8b95ac9d662e43 P 19fb518a6ede476f809c743054941e97: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/89392b248b7746d5bf8b95ac9d662e43
I20240503 08:12:35.610992 26948 catalog_manager.cc:4830] TS 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395): tablet 89392b248b7746d5bf8b95ac9d662e43 (table rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]) successfully deleted
I20240503 08:12:35.655340 27230 consensus_queue.cc:237] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 471, Committed index: 471, Last appended: 1.471, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:35.657704 27624 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 471. Preceding OpId from leader: term: 1 index: 472. (index mismatch)
I20240503 08:12:35.657831 28746 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 471. Preceding OpId from leader: term: 1 index: 472. (index mismatch)
W20240503 08:12:35.658268 27169 consensus_peers.cc:485] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c364e9ddad74e8e882df9080d9d5372. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:35.658497 27749 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 471. Preceding OpId from leader: term: 1 index: 472. (index mismatch)
I20240503 08:12:35.658551 31087 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 472, Last known committed idx: 471, Time since last communication: 0.000s
I20240503 08:12:35.659219 27491 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 471. Preceding OpId from leader: term: 1 index: 472. (index mismatch)
I20240503 08:12:35.659668 30964 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 472, Last known committed idx: 471, Time since last communication: 0.000s
I20240503 08:12:35.659972 31136 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 472, Last known committed idx: 471, Time since last communication: 0.000s
I20240503 08:12:35.660184 31087 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 472, Last known committed idx: 471, Time since last communication: 0.000s
I20240503 08:12:35.663468 30964 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Committing config change with OpId 1.472: config changed from index 125 to 472, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.663944 27491 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.472: config changed from index 125 to 472, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.667045 28750 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.472: config changed from index 125 to 472, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.668731 26959 catalog_manager.cc:5484] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 reported cstate change: config changed from index 125 to 472, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.672402 27749 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.472: config changed from index 125 to 472, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.680066 27624 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.472: config changed from index 125 to 472, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.694998 27097 consensus_queue.cc:237] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 442, Committed index: 442, Last appended: 1.442, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:35.697399 28750 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240503 08:12:35.697660 27749 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240503 08:12:35.697661 27361 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240503 08:12:35.697865 27491 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240503 08:12:35.698091 31212 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240503 08:12:35.698320 31202 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
W20240503 08:12:35.698629 27038 consensus_peers.cc:485] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f94c738c0f34d04bd933cc09c05d6a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:35.698956 31277 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240503 08:12:35.699273 31212 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240503 08:12:35.711871 31212 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.443: config changed from index 189 to 443, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.713222 28750 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 189 to 443, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.713222 27361 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 189 to 443, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.713725 27749 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 189 to 443, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.717378 27491 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 189 to 443, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.718300 26954 catalog_manager.cc:5484] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 189 to 443, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:35.933897 31307 ts_tablet_manager.cc:902] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
I20240503 08:12:35.937362 31307 tablet_copy_client.cc:308] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:41395
I20240503 08:12:35.938756 27515 tablet_copy_service.cc:140] P 19fb518a6ede476f809c743054941e97: Received BeginTabletCopySession request for tablet 03b4df9c9c0c4ec59b00338de1ae2920 from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:51009)
I20240503 08:12:35.938932 27515 tablet_copy_service.cc:161] P 19fb518a6ede476f809c743054941e97: Beginning new tablet copy session on tablet 03b4df9c9c0c4ec59b00338de1ae2920 from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:51009: session id = b4654488aec741f299e533fdf3715ce7-03b4df9c9c0c4ec59b00338de1ae2920
I20240503 08:12:35.940866 27515 tablet_copy_source_session.cc:205] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:35.941992 31307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b4df9c9c0c4ec59b00338de1ae2920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.946401 31307 tablet_copy_client.cc:783] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:35.946646 31307 tablet_copy_client.cc:647] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:35.949296 31307 tablet_copy_client.cc:523] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:35.951918 27624 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 4 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240503 08:12:35.952463 31077 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:35.952621 31307 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:35.953994 27361 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 4 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240503 08:12:35.954631 31077 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:35.966970 31312 ts_tablet_manager.cc:902] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:12:36.000119 31312 tablet_copy_client.cc:308] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39261
I20240503 08:12:36.024906 27124 tablet_copy_service.cc:140] P f4d3aef6d5624cf4adc3859dbadccb80: Received BeginTabletCopySession request for tablet e8c93cddbbf84976baa817ab0d98a811 from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:44451)
I20240503 08:12:36.025065 27124 tablet_copy_service.cc:161] P f4d3aef6d5624cf4adc3859dbadccb80: Beginning new tablet copy session on tablet e8c93cddbbf84976baa817ab0d98a811 from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:44451: session id = b4654488aec741f299e533fdf3715ce7-e8c93cddbbf84976baa817ab0d98a811
I20240503 08:12:36.028004 27124 tablet_copy_source_session.cc:205] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:36.029342 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c93cddbbf84976baa817ab0d98a811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.036022 31312 tablet_copy_client.cc:783] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:36.036391 31312 tablet_copy_client.cc:647] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:36.039321 31312 tablet_copy_client.cc:523] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:36.039546 30761 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 4 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240503 08:12:36.040210 31078 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:36.042937 31312 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:36.067126 27230 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 4 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240503 08:12:36.069023 28750 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 4 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240503 08:12:36.069535 31077 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:36.072832 31077 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:36.185479 31344 ts_tablet_manager.cc:902] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503)
I20240503 08:12:36.258049 31344 tablet_copy_client.cc:308] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:33503
I20240503 08:12:36.258785 27255 tablet_copy_service.cc:140] P 36fc7a106457453ba91f05b78a5d342b: Received BeginTabletCopySession request for tablet 4c364e9ddad74e8e882df9080d9d5372 from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:60703)
I20240503 08:12:36.258961 27255 tablet_copy_service.cc:161] P 36fc7a106457453ba91f05b78a5d342b: Beginning new tablet copy session on tablet 4c364e9ddad74e8e882df9080d9d5372 from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:60703: session id = b4654488aec741f299e533fdf3715ce7-4c364e9ddad74e8e882df9080d9d5372
I20240503 08:12:36.261601 27255 tablet_copy_source_session.cc:205] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:36.267055 31354 ts_tablet_manager.cc:902] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:12:36.282706 31344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c364e9ddad74e8e882df9080d9d5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.289532 31344 tablet_copy_client.cc:783] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:36.289860 31344 tablet_copy_client.cc:647] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:36.293340 31344 tablet_copy_client.cc:523] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:36.297187 31344 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:36.312538 31354 tablet_copy_client.cc:308] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39261
I20240503 08:12:36.313282 27124 tablet_copy_service.cc:140] P f4d3aef6d5624cf4adc3859dbadccb80: Received BeginTabletCopySession request for tablet 8f94c738c0f34d04bd933cc09c05d6a6 from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:44451)
I20240503 08:12:36.313428 27124 tablet_copy_service.cc:161] P f4d3aef6d5624cf4adc3859dbadccb80: Beginning new tablet copy session on tablet 8f94c738c0f34d04bd933cc09c05d6a6 from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:44451: session id = b4654488aec741f299e533fdf3715ce7-8f94c738c0f34d04bd933cc09c05d6a6
I20240503 08:12:36.316293 27124 tablet_copy_source_session.cc:205] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:36.317590 31354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f94c738c0f34d04bd933cc09c05d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.324558 31354 tablet_copy_client.cc:783] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:36.324858 31354 tablet_copy_client.cc:647] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:36.327714 31354 tablet_copy_client.cc:523] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:36.331389 31354 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:36.652603 27491 consensus_queue.cc:237] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 7, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:36.654784 28750 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 4 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240503 08:12:36.655201 27361 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 4 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240503 08:12:36.655447 31178 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:36.655753 31307 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=446 overwritten=0 applied=446 ignored=0} inserts{seen=1115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:36.655843 31077 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:36.656215 31307 tablet_bootstrap.cc:492] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:36.656572 31307 ts_tablet_manager.cc:1367] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.704s user 0.494s sys 0.074s
I20240503 08:12:36.657364 31307 raft_consensus.cc:348] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:36.657685 31307 raft_consensus.cc:729] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:36.657938 31307 consensus_queue.cc:260] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 446, Last appended: 1.446, Last appended by leader: 446, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 446 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:36.659504 27624 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 4 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240503 08:12:36.660008 30761 raft_consensus.cc:1261] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 4 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20240503 08:12:36.662736 31077 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:36.663161 31077 consensus_queue.cc:1035] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:36.666859 31178 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 [term 4 LEADER]: Committing config change with OpId 4.9: config changed from index 7 to 9, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:36.672093 31307 ts_tablet_manager.cc:1398] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20240503 08:12:36.672180 27361 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P fe8fdcdebbff4703a924f83646531115 [term 4 FOLLOWER]: Committing config change with OpId 4.9: config changed from index 7 to 9, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:36.675421 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8d5c3e2f8e6428aa457ba21d536cea5 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:36.675890 27515 tablet_copy_service.cc:342] P 19fb518a6ede476f809c743054941e97: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-03b4df9c9c0c4ec59b00338de1ae2920 received from {username='slave'} at 127.24.123.8:51009
I20240503 08:12:36.676013 27515 tablet_copy_service.cc:433] P 19fb518a6ede476f809c743054941e97: ending tablet copy session b4654488aec741f299e533fdf3715ce7-03b4df9c9c0c4ec59b00338de1ae2920 on tablet 03b4df9c9c0c4ec59b00338de1ae2920 with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:36.676703 30761 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 [term 4 FOLLOWER]: Committing config change with OpId 4.9: config changed from index 7 to 9, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:36.681012 28750 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 4 FOLLOWER]: Committing config change with OpId 4.9: config changed from index 7 to 9, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:36.686607 27624 raft_consensus.cc:2932] T b8d5c3e2f8e6428aa457ba21d536cea5 P 2de33eb4315b4547889d1e6edae2f2dd [term 4 FOLLOWER]: Committing config change with OpId 4.9: config changed from index 7 to 9, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:36.689529 26958 catalog_manager.cc:5484] T b8d5c3e2f8e6428aa457ba21d536cea5 P b4654488aec741f299e533fdf3715ce7 reported cstate change: config changed from index 7 to 9, VOTER 36fc7a106457453ba91f05b78a5d342b (127.24.123.2) evicted. New cstate: current_term: 4 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:36.730301 27213 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d5c3e2f8e6428aa457ba21d536cea5 with delete_type TABLET_DATA_TOMBSTONED (TS 36fc7a106457453ba91f05b78a5d342b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50738
I20240503 08:12:36.748096 31397 tablet_replica.cc:330] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: stopping tablet replica
I20240503 08:12:36.748291 31397 raft_consensus.cc:2229] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:36.748441 31397 raft_consensus.cc:2258] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:36.752326 31397 ts_tablet_manager.cc:1875] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:36.758605 31397 ts_tablet_manager.cc:1888] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.8
I20240503 08:12:36.758785 31397 log.cc:1196] T b8d5c3e2f8e6428aa457ba21d536cea5 P 36fc7a106457453ba91f05b78a5d342b: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8d5c3e2f8e6428aa457ba21d536cea5
I20240503 08:12:36.759897 26947 catalog_manager.cc:4830] TS 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): tablet b8d5c3e2f8e6428aa457ba21d536cea5 (table TestTable [id=0008359f803b498b9c188b0fffbee30e]) successfully deleted
I20240503 08:12:36.768029 31312 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=350 overwritten=0 applied=350 ignored=0} inserts{seen=1019 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:36.768638 31312 tablet_bootstrap.cc:492] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:36.769038 31312 ts_tablet_manager.cc:1367] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.726s user 0.361s sys 0.061s
I20240503 08:12:36.769946 31312 raft_consensus.cc:348] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:36.770395 31312 raft_consensus.cc:729] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:36.770648 31312 consensus_queue.cc:260] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 350, Last appended: 1.350, Last appended by leader: 350, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:36.771550 31312 ts_tablet_manager.cc:1398] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:36.772207 27124 tablet_copy_service.cc:342] P f4d3aef6d5624cf4adc3859dbadccb80: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-e8c93cddbbf84976baa817ab0d98a811 received from {username='slave'} at 127.24.123.8:44451
I20240503 08:12:36.772327 27124 tablet_copy_service.cc:433] P f4d3aef6d5624cf4adc3859dbadccb80: ending tablet copy session b4654488aec741f299e533fdf3715ce7-e8c93cddbbf84976baa817ab0d98a811 on tablet e8c93cddbbf84976baa817ab0d98a811 with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:36.974241 31344 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=472 overwritten=0 applied=472 ignored=0} inserts{seen=1141 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:36.974670 31344 tablet_bootstrap.cc:492] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:36.975056 31344 ts_tablet_manager.cc:1367] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.678s user 0.477s sys 0.085s
I20240503 08:12:36.975823 31344 raft_consensus.cc:348] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:36.976171 31344 raft_consensus.cc:729] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:36.976392 31344 consensus_queue.cc:260] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 472, Last appended: 1.472, Last appended by leader: 472, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:36.977092 31344 ts_tablet_manager.cc:1398] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:36.977948 27254 tablet_copy_service.cc:342] P 36fc7a106457453ba91f05b78a5d342b: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-4c364e9ddad74e8e882df9080d9d5372 received from {username='slave'} at 127.24.123.8:60703
I20240503 08:12:36.978065 27254 tablet_copy_service.cc:433] P 36fc7a106457453ba91f05b78a5d342b: ending tablet copy session b4654488aec741f299e533fdf3715ce7-4c364e9ddad74e8e882df9080d9d5372 on tablet 4c364e9ddad74e8e882df9080d9d5372 with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:36.988366 31354 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=443 overwritten=0 applied=443 ignored=0} inserts{seen=1125 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:36.988818 31354 tablet_bootstrap.cc:492] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:36.989183 31354 ts_tablet_manager.cc:1367] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.658s user 0.473s sys 0.050s
I20240503 08:12:36.989904 31354 raft_consensus.cc:348] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:36.990145 31354 raft_consensus.cc:729] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:36.990341 31354 consensus_queue.cc:260] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 443, Last appended: 1.443, Last appended by leader: 443, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:36.990821 31354 ts_tablet_manager.cc:1398] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:36.991544 27124 tablet_copy_service.cc:342] P f4d3aef6d5624cf4adc3859dbadccb80: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-8f94c738c0f34d04bd933cc09c05d6a6 received from {username='slave'} at 127.24.123.8:44451
I20240503 08:12:36.991693 27124 tablet_copy_service.cc:433] P f4d3aef6d5624cf4adc3859dbadccb80: ending tablet copy session b4654488aec741f299e533fdf3715ce7-8f94c738c0f34d04bd933cc09c05d6a6 on tablet 8f94c738c0f34d04bd933cc09c05d6a6 with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:37.079301 30761 raft_consensus.cc:1203] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.445->[1.446-1.446] Dedup: 1.446->[]
I20240503 08:12:37.140355 30761 raft_consensus.cc:1203] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.349->[1.350-1.350] Dedup: 1.350->[]
I20240503 08:12:37.336827 30761 raft_consensus.cc:1203] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.471->[1.472-1.472] Dedup: 1.472->[]
I20240503 08:12:37.421166 30761 raft_consensus.cc:1203] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.442->[1.443-1.443] Dedup: 1.443->[]
I20240503 08:12:37.624068 31465 raft_consensus.cc:1050] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:37.624812 31465 consensus_queue.cc:237] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:37.627020 30761 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240503 08:12:37.627000 27624 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240503 08:12:37.627616 27230 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240503 08:12:37.627583 31410 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 08:12:37.627665 27491 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240503 08:12:37.628089 31410 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 08:12:37.628326 31212 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 08:12:37.628484 28750 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240503 08:12:37.628955 31487 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 08:12:37.629995 31410 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240503 08:12:37.632408 31465 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.351: config changed from index 350 to 351, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.633139 27491 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.633212 28750 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.634047 27230 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.637388 27624 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.639318 26960 catalog_manager.cc:5484] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 reported cstate change: config changed from index 350 to 351, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.639482 30762 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } attrs { replace: true } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.651571 27097 consensus_queue.cc:237] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 1.351, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:37.654518 30761 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20240503 08:12:37.654831 27230 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20240503 08:12:37.655362 31487 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 08:12:37.656051 27491 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20240503 08:12:37.655998 31487 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 08:12:37.657092 31487 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 08:12:37.657711 31497 raft_consensus.cc:1050] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:37.658636 31497 consensus_queue.cc:237] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 446, Committed index: 446, Last appended: 1.446, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:37.658663 31465 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.352: config changed from index 351 to 352, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.659126 27230 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.662103 28746 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240503 08:12:37.662401 26950 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e8c93cddbbf84976baa817ab0d98a811 with cas_config_opid_index 351: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:37.662962 31078 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240503 08:12:37.663511 27624 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240503 08:12:37.663411 30761 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.664940 31078 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240503 08:12:37.664703 26955 catalog_manager.cc:5484] T e8c93cddbbf84976baa817ab0d98a811 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: config changed from index 351 to 352, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.664748 27491 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.668723 28750 raft_consensus.cc:1261] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20240503 08:12:37.670188 27605 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c93cddbbf84976baa817ab0d98a811 with delete_type TABLET_DATA_TOMBSTONED (TS 2de33eb4315b4547889d1e6edae2f2dd not found in new config with opid_index 352) from {username='slave'} at 127.0.0.1:33064
I20240503 08:12:37.670758 31504 tablet_replica.cc:330] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd: stopping tablet replica
I20240503 08:12:37.670823 31465 consensus_queue.cc:1035] T e8c93cddbbf84976baa817ab0d98a811 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 08:12:37.671028 31504 raft_consensus.cc:2229] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:37.671378 31504 raft_consensus.cc:2258] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:37.673781 31504 ts_tablet_manager.cc:1875] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:37.673704 28750 raft_consensus.cc:2932] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, VOTER 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.679816 27749 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240503 08:12:37.679863 30761 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240503 08:12:37.680356 31496 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240503 08:12:37.680678 31078 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240503 08:12:37.681191 31504 ts_tablet_manager.cc:1888] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.351
I20240503 08:12:37.681465 31504 log.cc:1196] T e8c93cddbbf84976baa817ab0d98a811 P 2de33eb4315b4547889d1e6edae2f2dd: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/wals/e8c93cddbbf84976baa817ab0d98a811
I20240503 08:12:37.681814 27230 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 1 index: 447. (index mismatch)
I20240503 08:12:37.682657 31498 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240503 08:12:37.682727 26948 catalog_manager.cc:4830] TS 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185): tablet e8c93cddbbf84976baa817ab0d98a811 (table rebalance_test_table_2 [id=225aa297be6640d8b676ce90a19a1ba4]) successfully deleted
I20240503 08:12:37.685077 31496 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.447: config changed from index 446 to 447, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.685659 27230 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.685699 30761 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.690515 26954 catalog_manager.cc:5484] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7 reported cstate change: config changed from index 446 to 447, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.691601 28746 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.691637 27624 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.691758 27749 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.447: config changed from index 446 to 447, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 447 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false replace: true } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.699312 27491 consensus_queue.cc:237] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 447, Committed index: 447, Last appended: 1.447, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:37.701157 27749 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240503 08:12:37.701421 30761 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240503 08:12:37.701406 27230 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240503 08:12:37.701716 31498 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240503 08:12:37.702143 31496 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240503 08:12:37.702396 31497 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240503 08:12:37.703572 27624 raft_consensus.cc:1261] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 19fb518a6ede476f809c743054941e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240503 08:12:37.704093 31498 consensus_queue.cc:1035] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240503 08:12:37.705165 31498 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 [term 1 LEADER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.705662 27230 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.706969 27751 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.707407 27624 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.709524 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03b4df9c9c0c4ec59b00338de1ae2920 with cas_config_opid_index 447: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:37.710918 26958 catalog_manager.cc:5484] T 03b4df9c9c0c4ec59b00338de1ae2920 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: config changed from index 447 to 448, VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) evicted. New cstate: current_term: 1 leader_uuid: "19fb518a6ede476f809c743054941e97" committed_config { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.714201 30761 raft_consensus.cc:2932] T 03b4df9c9c0c4ec59b00338de1ae2920 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 447 to 448, VOTER 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.724210 28730 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b4df9c9c0c4ec59b00338de1ae2920 with delete_type TABLET_DATA_TOMBSTONED (TS 1980d01e2e3e4840a8ddfca4d4e366b2 not found in new config with opid_index 448) from {username='slave'} at 127.0.0.1:35818
I20240503 08:12:37.724987 31513 tablet_replica.cc:330] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: stopping tablet replica
I20240503 08:12:37.725361 31513 raft_consensus.cc:2229] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:37.725615 31513 raft_consensus.cc:2258] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:37.727733 31513 ts_tablet_manager.cc:1875] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:37.732316 31513 ts_tablet_manager.cc:1888] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.447
I20240503 08:12:37.732437 31513 log.cc:1196] T 03b4df9c9c0c4ec59b00338de1ae2920 P 1980d01e2e3e4840a8ddfca4d4e366b2: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/wals/03b4df9c9c0c4ec59b00338de1ae2920
I20240503 08:12:37.733325 26950 catalog_manager.cc:4830] TS 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167): tablet 03b4df9c9c0c4ec59b00338de1ae2920 (table rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]) successfully deleted
I20240503 08:12:37.811070 31465 raft_consensus.cc:1050] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:37.812111 31465 consensus_queue.cc:237] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 443, Committed index: 443, Last appended: 1.443, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:37.814790 27363 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 08:12:37.814829 27751 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 08:12:37.815407 28746 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 08:12:37.815762 31465 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 08:12:37.816076 31487 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 08:12:37.816542 31487 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 08:12:37.816648 30761 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 08:12:37.817193 27491 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 08:12:37.819346 31487 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 08:12:37.819831 31487 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 08:12:37.830746 31487 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.444: config changed from index 443 to 444, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.832356 28746 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.834453 27491 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.835148 30761 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.839254 27363 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.839733 27751 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.840871 26958 catalog_manager.cc:5484] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 reported cstate change: config changed from index 443 to 444, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:37.847260 27097 consensus_queue.cc:237] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 444, Committed index: 444, Last appended: 1.444, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:37.849850 30761 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240503 08:12:37.850292 27363 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240503 08:12:37.850378 31535 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240503 08:12:37.850838 31487 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240503 08:12:37.851809 28746 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240503 08:12:37.853315 31535 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 LEADER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.854066 27491 raft_consensus.cc:1261] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer f4d3aef6d5624cf4adc3859dbadccb80: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240503 08:12:37.853935 27361 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.854604 31487 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240503 08:12:37.856083 27491 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.857245 31410 consensus_queue.cc:1035] T 8f94c738c0f34d04bd933cc09c05d6a6 P f4d3aef6d5624cf4adc3859dbadccb80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240503 08:12:37.858069 26950 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f94c738c0f34d04bd933cc09c05d6a6 with cas_config_opid_index 444: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:37.858175 30761 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.859934 26955 catalog_manager.cc:5484] T 8f94c738c0f34d04bd933cc09c05d6a6 P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 444 to 445, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" committed_config { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.859310 28746 raft_consensus.cc:2932] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.871027 27733 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f94c738c0f34d04bd933cc09c05d6a6 with delete_type TABLET_DATA_TOMBSTONED (TS d8d7c32cb9a241c99853a6639451c4ed not found in new config with opid_index 445) from {username='slave'} at 127.0.0.1:42730
I20240503 08:12:37.872354 31546 tablet_replica.cc:330] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed: stopping tablet replica
I20240503 08:12:37.872547 31546 raft_consensus.cc:2229] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:37.872809 31546 raft_consensus.cc:2258] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:37.874472 31546 ts_tablet_manager.cc:1875] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:37.880904 31546 ts_tablet_manager.cc:1888] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.444
I20240503 08:12:37.881057 31546 log.cc:1196] T 8f94c738c0f34d04bd933cc09c05d6a6 P d8d7c32cb9a241c99853a6639451c4ed: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/8f94c738c0f34d04bd933cc09c05d6a6
I20240503 08:12:37.881788 26950 catalog_manager.cc:4830] TS d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465): tablet 8f94c738c0f34d04bd933cc09c05d6a6 (table rebalance_test_table_0 [id=4a18e128d36f451cbe424b48c835c4e2]) successfully deleted
I20240503 08:12:37.931587 31557 raft_consensus.cc:1050] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:37.932389 31557 consensus_queue.cc:237] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 472, Committed index: 472, Last appended: 1.472, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:37.935990 28746 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 472. Preceding OpId from leader: term: 1 index: 473. (index mismatch)
I20240503 08:12:37.936506 31560 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 473, Last known committed idx: 472, Time since last communication: 0.000s
I20240503 08:12:37.936729 27751 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 472. Preceding OpId from leader: term: 1 index: 473. (index mismatch)
I20240503 08:12:37.937376 31560 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 473, Last known committed idx: 472, Time since last communication: 0.000s
I20240503 08:12:37.939782 30761 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 472. Preceding OpId from leader: term: 1 index: 473. (index mismatch)
I20240503 08:12:37.940317 27624 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 472. Preceding OpId from leader: term: 1 index: 473. (index mismatch)
I20240503 08:12:37.940805 31560 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 473, Last known committed idx: 472, Time since last communication: 0.000s
I20240503 08:12:37.941252 31464 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 473, Last known committed idx: 472, Time since last communication: 0.000s
I20240503 08:12:37.947326 31560 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Committing config change with OpId 1.473: config changed from index 472 to 473, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.952019 27624 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.473: config changed from index 472 to 473, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.958070 26961 catalog_manager.cc:5484] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd reported cstate change: config changed from index 472 to 473, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.963439 27491 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 472. Preceding OpId from leader: term: 1 index: 473. (index mismatch)
I20240503 08:12:37.964043 31558 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 473, Last known committed idx: 472, Time since last communication: 0.000s
I20240503 08:12:37.964099 28746 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.473: config changed from index 472 to 473, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.964320 27751 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.473: config changed from index 472 to 473, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.972194 27491 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.473: config changed from index 472 to 473, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.977787 30761 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.473: config changed from index 472 to 473, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 473 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } attrs { replace: true } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.979939 27230 consensus_queue.cc:237] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 473, Committed index: 473, Last appended: 1.473, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:37.982381 27624 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 473. Preceding OpId from leader: term: 1 index: 474. (index mismatch)
I20240503 08:12:37.982726 28746 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 473. Preceding OpId from leader: term: 1 index: 474. (index mismatch)
I20240503 08:12:37.982916 31557 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 474, Last known committed idx: 473, Time since last communication: 0.000s
I20240503 08:12:37.983043 27751 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 473. Preceding OpId from leader: term: 1 index: 474. (index mismatch)
I20240503 08:12:37.983359 31557 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 474, Last known committed idx: 473, Time since last communication: 0.000s
I20240503 08:12:37.983628 31561 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 474, Last known committed idx: 473, Time since last communication: 0.000s
I20240503 08:12:37.985905 31561 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [term 1 LEADER]: Committing config change with OpId 1.474: config changed from index 473 to 474, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.986433 28746 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.474: config changed from index 473 to 474, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.986424 27751 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.474: config changed from index 473 to 474, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.990475 26947 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c364e9ddad74e8e882df9080d9d5372 with cas_config_opid_index 473: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:37.991533 30760 raft_consensus.cc:1261] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Refusing update from remote peer 36fc7a106457453ba91f05b78a5d342b: Log matching property violated. Preceding OpId in replica: term: 1 index: 473. Preceding OpId from leader: term: 1 index: 474. (index mismatch)
I20240503 08:12:37.992610 31558 consensus_queue.cc:1035] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 474, Last known committed idx: 473, Time since last communication: 0.000s
I20240503 08:12:37.994179 30760 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.474: config changed from index 473 to 474, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.995308 27624 raft_consensus.cc:2932] T 4c364e9ddad74e8e882df9080d9d5372 P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Committing config change with OpId 1.474: config changed from index 473 to 474, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New config: { opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:37.998052 26961 catalog_manager.cc:5484] T 4c364e9ddad74e8e882df9080d9d5372 P 36fc7a106457453ba91f05b78a5d342b reported cstate change: config changed from index 473 to 474, VOTER 19fb518a6ede476f809c743054941e97 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "36fc7a106457453ba91f05b78a5d342b" committed_config { opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:38.003780 27473 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c364e9ddad74e8e882df9080d9d5372 with delete_type TABLET_DATA_TOMBSTONED (TS 19fb518a6ede476f809c743054941e97 not found in new config with opid_index 474) from {username='slave'} at 127.0.0.1:56480
I20240503 08:12:38.004434 31567 tablet_replica.cc:330] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97: stopping tablet replica
I20240503 08:12:38.004623 31567 raft_consensus.cc:2229] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:38.004904 31567 raft_consensus.cc:2258] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:38.015578 31567 ts_tablet_manager.cc:1875] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:38.020119 31567 ts_tablet_manager.cc:1888] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.473
I20240503 08:12:38.020238 31567 log.cc:1196] T 4c364e9ddad74e8e882df9080d9d5372 P 19fb518a6ede476f809c743054941e97: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c364e9ddad74e8e882df9080d9d5372
I20240503 08:12:38.020920 26948 catalog_manager.cc:4830] TS 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395): tablet 4c364e9ddad74e8e882df9080d9d5372 (table rebalance_test_table_4 [id=53ec6f491e9d4c4ba45ce9e65edfcd3c]) successfully deleted
I20240503 08:12:38.224445 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:38.225261 27473 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:38.236399 27213 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:38.311904 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:38.327431 30742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:38.355382 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:38.356253 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:38.365149 27344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:38.441015 27361 consensus_queue.cc:237] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 1.305, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:38.444537 28746 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 305. Preceding OpId from leader: term: 1 index: 306. (index mismatch)
W20240503 08:12:38.444694 27299 consensus_peers.cc:485] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 -> Peer b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727): Couldn't send request to peer b4654488aec741f299e533fdf3715ce7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 921b4706d2bc4693ae5a0a10b310d5d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:38.445096 27097 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 305. Preceding OpId from leader: term: 1 index: 306. (index mismatch)
I20240503 08:12:38.445096 31500 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20240503 08:12:38.445647 31616 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20240503 08:12:38.446223 27751 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 305. Preceding OpId from leader: term: 1 index: 306. (index mismatch)
I20240503 08:12:38.447021 31500 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20240503 08:12:38.449501 31500 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.306: config changed from index 84 to 306, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:38.452493 27751 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.306: config changed from index 84 to 306, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:38.458858 26958 catalog_manager.cc:5484] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed reported cstate change: config changed from index 84 to 306, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:38.462986 28746 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.306: config changed from index 84 to 306, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:38.463717 27491 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 305. Preceding OpId from leader: term: 1 index: 306. (index mismatch)
I20240503 08:12:38.464488 31616 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20240503 08:12:38.472031 27097 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.306: config changed from index 84 to 306, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:38.479056 27491 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.306: config changed from index 84 to 306, NON_VOTER b4654488aec741f299e533fdf3715ce7 (127.24.123.8) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } } }
I20240503 08:12:38.920434 31642 ts_tablet_manager.cc:902] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: Initiating tablet copy from peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977)
I20240503 08:12:38.921445 31642 tablet_copy_client.cc:308] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:43977
I20240503 08:12:38.922005 27385 tablet_copy_service.cc:140] P fe8fdcdebbff4703a924f83646531115: Received BeginTabletCopySession request for tablet 921b4706d2bc4693ae5a0a10b310d5d4 from peer b4654488aec741f299e533fdf3715ce7 ({username='slave'} at 127.24.123.8:36441)
I20240503 08:12:38.922142 27385 tablet_copy_service.cc:161] P fe8fdcdebbff4703a924f83646531115: Beginning new tablet copy session on tablet 921b4706d2bc4693ae5a0a10b310d5d4 from peer b4654488aec741f299e533fdf3715ce7 at {username='slave'} at 127.24.123.8:36441: session id = b4654488aec741f299e533fdf3715ce7-921b4706d2bc4693ae5a0a10b310d5d4
I20240503 08:12:38.924024 27385 tablet_copy_source_session.cc:205] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:38.925197 31642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921b4706d2bc4693ae5a0a10b310d5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.931419 31642 tablet_copy_client.cc:783] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:38.931677 31642 tablet_copy_client.cc:647] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:38.933848 31642 tablet_copy_client.cc:523] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:38.936415 31642 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: Bootstrap starting.
I20240503 08:12:39.215576 31642 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=306 overwritten=0 applied=306 ignored=0} inserts{seen=988 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:39.215953 31642 tablet_bootstrap.cc:492] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: Bootstrap complete.
I20240503 08:12:39.216213 31642 ts_tablet_manager.cc:1367] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: Time spent bootstrapping tablet: real 0.280s user 0.262s sys 0.016s
I20240503 08:12:39.216802 31642 raft_consensus.cc:348] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:39.217031 31642 raft_consensus.cc:729] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Initialized, Role: LEARNER
I20240503 08:12:39.217206 31642 consensus_queue.cc:260] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 306, Last appended: 1.306, Last appended by leader: 306, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: true } }
I20240503 08:12:39.217875 31642 ts_tablet_manager.cc:1398] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:39.218643 27385 tablet_copy_service.cc:342] P fe8fdcdebbff4703a924f83646531115: Request end of tablet copy session b4654488aec741f299e533fdf3715ce7-921b4706d2bc4693ae5a0a10b310d5d4 received from {username='slave'} at 127.24.123.8:36441
I20240503 08:12:39.218763 27385 tablet_copy_service.cc:433] P fe8fdcdebbff4703a924f83646531115: ending tablet copy session b4654488aec741f299e533fdf3715ce7-921b4706d2bc4693ae5a0a10b310d5d4 on tablet 921b4706d2bc4693ae5a0a10b310d5d4 with peer b4654488aec741f299e533fdf3715ce7
I20240503 08:12:39.329370 30760 raft_consensus.cc:1203] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.305->[1.306-1.306] Dedup: 1.306->[]
I20240503 08:12:39.874503 31673 raft_consensus.cc:1050] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: attempting to promote NON_VOTER b4654488aec741f299e533fdf3715ce7 to VOTER
I20240503 08:12:39.875315 31673 consensus_queue.cc:237] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 1.306, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:39.877472 30760 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 1 LEARNER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 1 index: 307. (index mismatch)
I20240503 08:12:39.877641 28746 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 1 index: 307. (index mismatch)
I20240503 08:12:39.878106 31616 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20240503 08:12:39.878647 31675 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20240503 08:12:39.880553 27751 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 1 index: 307. (index mismatch)
I20240503 08:12:39.881098 31616 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20240503 08:12:39.892746 27097 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 1 index: 307. (index mismatch)
I20240503 08:12:39.893371 31675 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20240503 08:12:39.894316 27494 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Refusing update from remote peer fe8fdcdebbff4703a924f83646531115: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 1 index: 307. (index mismatch)
I20240503 08:12:39.894953 31676 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20240503 08:12:39.896646 31676 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Committing config change with OpId 1.307: config changed from index 306 to 307, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:39.899329 27751 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Committing config change with OpId 1.307: config changed from index 306 to 307, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:39.900610 27097 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Committing config change with OpId 1.307: config changed from index 306 to 307, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:39.902560 26954 catalog_manager.cc:5484] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 reported cstate change: config changed from index 306 to 307, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe8fdcdebbff4703a924f83646531115" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:39.902591 27494 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Committing config change with OpId 1.307: config changed from index 306 to 307, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:39.902900 28746 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Committing config change with OpId 1.307: config changed from index 306 to 307, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:39.904844 30760 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Committing config change with OpId 1.307: config changed from index 306 to 307, b4654488aec741f299e533fdf3715ce7 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:39.961815 27361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4"
dest_uuid: "fe8fdcdebbff4703a924f83646531115"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47320
I20240503 08:12:39.962039 27361 raft_consensus.cc:595] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:40.178318 27361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4"
dest_uuid: "fe8fdcdebbff4703a924f83646531115"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47336
I20240503 08:12:40.178494 27361 raft_consensus.cc:595] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:40.305198 31676 raft_consensus.cc:979] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: : Instructing follower b4654488aec741f299e533fdf3715ce7 to start an election
I20240503 08:12:40.305359 31676 raft_consensus.cc:1067] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Signalling peer b4654488aec741f299e533fdf3715ce7 to start an election
I20240503 08:12:40.306123 30760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4"
dest_uuid: "b4654488aec741f299e533fdf3715ce7"
from {username='slave'} at 127.24.123.3:42367
I20240503 08:12:40.306406 30760 raft_consensus.cc:482] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:40.306526 30760 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:40.308553 30760 raft_consensus.cc:504] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:40.310366 30760 leader_election.cc:288] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 election: Requested vote from peers fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167)
I20240503 08:12:40.310470 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "b4654488aec741f299e533fdf3715ce7" candidate_term: 2 candidate_status { last_received { term: 1 index: 307 } } ignore_live_leader: true dest_uuid: "fe8fdcdebbff4703a924f83646531115"
I20240503 08:12:40.310529 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "b4654488aec741f299e533fdf3715ce7" candidate_term: 2 candidate_status { last_received { term: 1 index: 307 } } ignore_live_leader: true dest_uuid: "19fb518a6ede476f809c743054941e97"
I20240503 08:12:40.310678 27361 raft_consensus.cc:3032] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:40.310688 27494 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:40.310751 27361 raft_consensus.cc:729] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe8fdcdebbff4703a924f83646531115, State: Running, Role: LEADER
I20240503 08:12:40.311187 27097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "b4654488aec741f299e533fdf3715ce7" candidate_term: 2 candidate_status { last_received { term: 1 index: 307 } } ignore_live_leader: true dest_uuid: "f4d3aef6d5624cf4adc3859dbadccb80"
I20240503 08:12:40.311367 27097 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:40.311419 27361 consensus_queue.cc:260] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 1.307, Last appended by leader: 307, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:40.312026 27361 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:40.313242 27097 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4654488aec741f299e533fdf3715ce7 in term 2.
I20240503 08:12:40.313302 27494 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4654488aec741f299e533fdf3715ce7 in term 2.
I20240503 08:12:40.314666 27361 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4654488aec741f299e533fdf3715ce7 in term 2.
I20240503 08:12:40.324823 30698 leader_election.cc:302] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 19fb518a6ede476f809c743054941e97, b4654488aec741f299e533fdf3715ce7, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115; no voters:
I20240503 08:12:40.325214 31685 raft_consensus.cc:2781] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:40.330233 27751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "b4654488aec741f299e533fdf3715ce7" candidate_term: 2 candidate_status { last_received { term: 1 index: 307 } } ignore_live_leader: true dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed"
I20240503 08:12:40.330333 31685 raft_consensus.cc:686] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 2 LEADER]: Becoming Leader. State: Replica: b4654488aec741f299e533fdf3715ce7, State: Running, Role: LEADER
I20240503 08:12:40.330451 27751 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:40.330971 28746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921b4706d2bc4693ae5a0a10b310d5d4" candidate_uuid: "b4654488aec741f299e533fdf3715ce7" candidate_term: 2 candidate_status { last_received { term: 1 index: 307 } } ignore_live_leader: true dest_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2"
I20240503 08:12:40.331128 28746 raft_consensus.cc:3037] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:40.331122 31685 consensus_queue.cc:237] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 1.307, Last appended by leader: 307, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:40.333096 27751 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4654488aec741f299e533fdf3715ce7 in term 2.
I20240503 08:12:40.333170 28746 raft_consensus.cc:2454] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4654488aec741f299e533fdf3715ce7 in term 2.
I20240503 08:12:40.335410 26959 catalog_manager.cc:5484] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 reported cstate change: term changed from 1 to 2, leader changed from fe8fdcdebbff4703a924f83646531115 (127.24.123.3) to b4654488aec741f299e533fdf3715ce7 (127.24.123.8). New cstate: current_term: 2 leader_uuid: "b4654488aec741f299e533fdf3715ce7" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:40.717264 27494 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Refusing update from remote peer b4654488aec741f299e533fdf3715ce7: Log matching property violated. Preceding OpId in replica: term: 1 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20240503 08:12:40.717869 31685 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20240503 08:12:40.779305 27361 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Refusing update from remote peer b4654488aec741f299e533fdf3715ce7: Log matching property violated. Preceding OpId in replica: term: 1 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20240503 08:12:40.779857 31685 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20240503 08:12:40.822607 27097 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 FOLLOWER]: Refusing update from remote peer b4654488aec741f299e533fdf3715ce7: Log matching property violated. Preceding OpId in replica: term: 1 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20240503 08:12:40.823099 31685 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20240503 08:12:40.826948 27751 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 2 FOLLOWER]: Refusing update from remote peer b4654488aec741f299e533fdf3715ce7: Log matching property violated. Preceding OpId in replica: term: 1 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20240503 08:12:40.827024 28746 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: Refusing update from remote peer b4654488aec741f299e533fdf3715ce7: Log matching property violated. Preceding OpId in replica: term: 1 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20240503 08:12:40.830242 31685 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20240503 08:12:40.830791 31733 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20240503 08:12:41.338466 30760 consensus_queue.cc:237] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 2.308, Last appended by leader: 307, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:41.340582 28746 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: Refusing update from remote peer b4654488aec741f299e533fdf3715ce7: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch)
I20240503 08:12:41.340714 27751 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 2 FOLLOWER]: Refusing update from remote peer b4654488aec741f299e533fdf3715ce7: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch)
I20240503 08:12:41.341324 27097 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 FOLLOWER]: Refusing update from remote peer b4654488aec741f299e533fdf3715ce7: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch)
I20240503 08:12:41.341831 31735 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.001s
I20240503 08:12:41.342123 31685 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20240503 08:12:41.342411 27494 raft_consensus.cc:1261] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Refusing update from remote peer b4654488aec741f299e533fdf3715ce7: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch)
I20240503 08:12:41.342362 31733 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20240503 08:12:41.342832 31735 consensus_queue.cc:1035] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20240503 08:12:41.345391 31735 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 [term 2 LEADER]: Committing config change with OpId 2.309: config changed from index 307 to 309, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:41.346720 27494 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 19fb518a6ede476f809c743054941e97 [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 307 to 309, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:41.347025 28746 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 307 to 309, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:41.349385 26948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 921b4706d2bc4693ae5a0a10b310d5d4 with cas_config_opid_index 307: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:41.350229 27751 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P d8d7c32cb9a241c99853a6639451c4ed [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 307 to 309, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:41.350512 27097 raft_consensus.cc:2932] T 921b4706d2bc4693ae5a0a10b310d5d4 P f4d3aef6d5624cf4adc3859dbadccb80 [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 307 to 309, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } } }
I20240503 08:12:41.351272 26960 catalog_manager.cc:5484] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 reported cstate change: config changed from index 307 to 309, VOTER fe8fdcdebbff4703a924f83646531115 (127.24.123.3) evicted. New cstate: current_term: 2 leader_uuid: "b4654488aec741f299e533fdf3715ce7" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:41.367245 27344 tablet_service.cc:1514] Processing DeleteTablet for tablet 921b4706d2bc4693ae5a0a10b310d5d4 with delete_type TABLET_DATA_TOMBSTONED (TS fe8fdcdebbff4703a924f83646531115 not found in new config with opid_index 309) from {username='slave'} at 127.0.0.1:60962
I20240503 08:12:41.369175 31761 tablet_replica.cc:330] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: stopping tablet replica
I20240503 08:12:41.369350 31761 raft_consensus.cc:2229] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.369506 31761 raft_consensus.cc:2258] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.370834 31761 ts_tablet_manager.cc:1875] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:41.374927 31761 ts_tablet_manager.cc:1888] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.308
I20240503 08:12:41.375044 31761 log.cc:1196] T 921b4706d2bc4693ae5a0a10b310d5d4 P fe8fdcdebbff4703a924f83646531115: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/921b4706d2bc4693ae5a0a10b310d5d4
I20240503 08:12:41.375654 26947 catalog_manager.cc:4830] TS fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): tablet 921b4706d2bc4693ae5a0a10b310d5d4 (table rebalance_test_table_3 [id=e89fa94dfc184007962228c3c499e7e5]) successfully deleted
I20240503 08:12:41.531491 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:41.537585 27473 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:41.584674 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:41.598322 27213 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:41.622963 30742 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:41.653725 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:41.654531 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:41.669510 27344 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:41.831715 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:41.841984 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:41.846570 27473 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:41.864351 30742 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:41.870960 27344 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:41.881942 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:41.887578 27213 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:41.903133 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:42.318464 28730 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:42.347182 30742 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:42.347527 27733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:42.355677 27213 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:42.356117 27605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:42.385715 27084 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:42.390846 27344 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:42.416875 27473 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
140b761e00e94e39b950d32863080556 | 127.24.123.62:40087 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.123.20:41695 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1980d01e2e3e4840a8ddfca4d4e366b2 | 127.24.123.7:41167 | HEALTHY | <none> | 0 | 0
19fb518a6ede476f809c743054941e97 | 127.24.123.4:41395 | HEALTHY | <none> | 4 | 0
2de33eb4315b4547889d1e6edae2f2dd | 127.24.123.5:43185 | HEALTHY | <none> | 4 | 0
36fc7a106457453ba91f05b78a5d342b | 127.24.123.2:33503 | HEALTHY | <none> | 2 | 0
b4654488aec741f299e533fdf3715ce7 | 127.24.123.8:35727 | HEALTHY | <none> | 1 | 0
d8d7c32cb9a241c99853a6639451c4ed | 127.24.123.6:43465 | HEALTHY | <none> | 0 | 0
f4d3aef6d5624cf4adc3859dbadccb80 | 127.24.123.1:39261 | HEALTHY | <none> | 3 | 0
fe8fdcdebbff4703a924f83646531115 | 127.24.123.3:43977 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.24.123.1 | experimental | 127.24.123.1:39261
local_ip_for_outbound_sockets | 127.24.123.2 | experimental | 127.24.123.2:33503
local_ip_for_outbound_sockets | 127.24.123.3 | experimental | 127.24.123.3:43977
local_ip_for_outbound_sockets | 127.24.123.4 | experimental | 127.24.123.4:41395
local_ip_for_outbound_sockets | 127.24.123.5 | experimental | 127.24.123.5:43185
local_ip_for_outbound_sockets | 127.24.123.6 | experimental | 127.24.123.6:43465
local_ip_for_outbound_sockets | 127.24.123.7 | experimental | 127.24.123.7:41167
local_ip_for_outbound_sockets | 127.24.123.8 | experimental | 127.24.123.8:35727
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.123.1:39261
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.123.2:33503
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.123.3:43977
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.24.123.4:41395
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.24.123.5:43185
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.24.123.6:43465
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.24.123.7:41167
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.24.123.8:35727
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.123.20:41695 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:12:42.489945 25068 log_verifier.cc:126] Checking tablet 03b4df9c9c0c4ec59b00338de1ae2920
I20240503 08:12:42.911896 25068 log_verifier.cc:177] Verified matching terms for 448 ops in tablet 03b4df9c9c0c4ec59b00338de1ae2920
I20240503 08:12:42.912952 25068 log_verifier.cc:126] Checking tablet 08b4767b5b4f4fb0bfc73c35a408a5ac
I20240503 08:12:43.279522 25068 log_verifier.cc:177] Verified matching terms for 422 ops in tablet 08b4767b5b4f4fb0bfc73c35a408a5ac
I20240503 08:12:43.280576 25068 log_verifier.cc:126] Checking tablet 43351c2d61024ae0afc96b6cf94d1741
I20240503 08:12:43.629256 25068 log_verifier.cc:177] Verified matching terms for 388 ops in tablet 43351c2d61024ae0afc96b6cf94d1741
I20240503 08:12:43.630713 25068 log_verifier.cc:126] Checking tablet 4c364e9ddad74e8e882df9080d9d5372
I20240503 08:12:44.079285 25068 log_verifier.cc:177] Verified matching terms for 474 ops in tablet 4c364e9ddad74e8e882df9080d9d5372
I20240503 08:12:44.080515 25068 log_verifier.cc:126] Checking tablet 89392b248b7746d5bf8b95ac9d662e43
I20240503 08:12:44.342826 25068 log_verifier.cc:177] Verified matching terms for 316 ops in tablet 89392b248b7746d5bf8b95ac9d662e43
I20240503 08:12:44.343629 25068 log_verifier.cc:126] Checking tablet 8e17a88a118b4ce591a266fd62b3d7a7
I20240503 08:12:44.791340 25068 log_verifier.cc:177] Verified matching terms for 522 ops in tablet 8e17a88a118b4ce591a266fd62b3d7a7
I20240503 08:12:44.792548 25068 log_verifier.cc:126] Checking tablet 8e4a8224c7c1430a9354e58bd16c7abd
I20240503 08:12:45.145095 25068 log_verifier.cc:177] Verified matching terms for 424 ops in tablet 8e4a8224c7c1430a9354e58bd16c7abd
I20240503 08:12:45.146077 25068 log_verifier.cc:126] Checking tablet 8f94c738c0f34d04bd933cc09c05d6a6
I20240503 08:12:45.521839 25068 log_verifier.cc:177] Verified matching terms for 445 ops in tablet 8f94c738c0f34d04bd933cc09c05d6a6
I20240503 08:12:45.522821 25068 log_verifier.cc:126] Checking tablet 921b4706d2bc4693ae5a0a10b310d5d4
I20240503 08:12:45.807929 25068 log_verifier.cc:177] Verified matching terms for 309 ops in tablet 921b4706d2bc4693ae5a0a10b310d5d4
I20240503 08:12:45.808940 25068 log_verifier.cc:126] Checking tablet a61402b1e1644ab7a0769b74e40e479b
I20240503 08:12:46.124828 25068 log_verifier.cc:177] Verified matching terms for 378 ops in tablet a61402b1e1644ab7a0769b74e40e479b
I20240503 08:12:46.125701 25068 log_verifier.cc:126] Checking tablet b8d5c3e2f8e6428aa457ba21d536cea5
I20240503 08:12:46.158244 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet b8d5c3e2f8e6428aa457ba21d536cea5
I20240503 08:12:46.158351 25068 log_verifier.cc:126] Checking tablet c42929dcee3c461c8db91346ca1ab0f5
I20240503 08:12:46.554306 25068 log_verifier.cc:177] Verified matching terms for 469 ops in tablet c42929dcee3c461c8db91346ca1ab0f5
I20240503 08:12:46.555354 25068 log_verifier.cc:126] Checking tablet ccb3a8dde7ec4d318b0961c8bad8032c
I20240503 08:12:46.993587 25068 log_verifier.cc:177] Verified matching terms for 498 ops in tablet ccb3a8dde7ec4d318b0961c8bad8032c
I20240503 08:12:46.995530 25068 log_verifier.cc:126] Checking tablet dd41f1db8043464583458568a5900a7d
I20240503 08:12:47.359068 25068 log_verifier.cc:177] Verified matching terms for 466 ops in tablet dd41f1db8043464583458568a5900a7d
I20240503 08:12:47.360111 25068 log_verifier.cc:126] Checking tablet e8c93cddbbf84976baa817ab0d98a811
I20240503 08:12:47.675143 25068 log_verifier.cc:177] Verified matching terms for 352 ops in tablet e8c93cddbbf84976baa817ab0d98a811
I20240503 08:12:47.675912 25068 log_verifier.cc:126] Checking tablet f09f0689f0714d8c8f45e24bd9adb533
I20240503 08:12:47.921896 25068 log_verifier.cc:177] Verified matching terms for 279 ops in tablet f09f0689f0714d8c8f45e24bd9adb533
I20240503 08:12:47.931596 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 27022
I20240503 08:12:47.954566 27144 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:48.234313 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 27022
W20240503 08:12:48.280285 30698 connection.cc:535] client connection to 127.24.123.1:39261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:48.280356 27171 connection.cc:535] client connection to 127.24.123.1:39261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:48.280542 27171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:48.280589 30698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:48.280934 27561 connection.cc:535] client connection to 127.24.123.1:39261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:48.281175 27561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:48.281670 30697 connection.cc:535] server connection from 127.24.123.1:51265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:48.281824 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 27153
W20240503 08:12:48.293311 27561 consensus_peers.cc:485] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd -> Peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Couldn't send request to peer f4d3aef6d5624cf4adc3859dbadccb80. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.293569 27561 consensus_peers.cc:485] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd -> Peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Couldn't send request to peer f4d3aef6d5624cf4adc3859dbadccb80. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.296067 27171 consensus_peers.cc:485] T 8e4a8224c7c1430a9354e58bd16c7abd P 36fc7a106457453ba91f05b78a5d342b -> Peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Couldn't send request to peer f4d3aef6d5624cf4adc3859dbadccb80. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.296873 30698 consensus_peers.cc:485] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 -> Peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Couldn't send request to peer f4d3aef6d5624cf4adc3859dbadccb80. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:48.298718 27274 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:12:48.317485 27301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111)
W20240503 08:12:48.318686 27301 consensus_peers.cc:485] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 -> Peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Couldn't send request to peer f4d3aef6d5624cf4adc3859dbadccb80. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.331692 27561 consensus_peers.cc:485] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd -> Peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Couldn't send request to peer f4d3aef6d5624cf4adc3859dbadccb80. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.488775 27301 consensus_peers.cc:485] T ccb3a8dde7ec4d318b0961c8bad8032c P fe8fdcdebbff4703a924f83646531115 -> Peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Couldn't send request to peer f4d3aef6d5624cf4adc3859dbadccb80. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:48.568202 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 27153
W20240503 08:12:48.606611 27298 connection.cc:535] client connection to 127.24.123.2:33503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:48.606962 27558 connection.cc:535] client connection to 127.24.123.2:33503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:48.607312 27428 connection.cc:535] client connection to 127.24.123.2:33503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:48.607533 27428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:48.607605 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 27283
W20240503 08:12:48.608088 27298 consensus_peers.cc:485] T 8e17a88a118b4ce591a266fd62b3d7a7 P fe8fdcdebbff4703a924f83646531115 -> Peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Couldn't send request to peer 36fc7a106457453ba91f05b78a5d342b. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.608578 27558 consensus_peers.cc:485] T dd41f1db8043464583458568a5900a7d P 2de33eb4315b4547889d1e6edae2f2dd -> Peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Couldn't send request to peer 36fc7a106457453ba91f05b78a5d342b. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.609308 27428 consensus_peers.cc:485] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 -> Peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Couldn't send request to peer 36fc7a106457453ba91f05b78a5d342b. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.609406 27428 consensus_peers.cc:485] T 03b4df9c9c0c4ec59b00338de1ae2920 P 19fb518a6ede476f809c743054941e97 -> Peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Couldn't send request to peer 36fc7a106457453ba91f05b78a5d342b. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:48.613332 27404 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:12:48.665798 27558 consensus_peers.cc:485] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd -> Peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Couldn't send request to peer 36fc7a106457453ba91f05b78a5d342b. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.824609 27428 consensus_peers.cc:485] T f09f0689f0714d8c8f45e24bd9adb533 P 19fb518a6ede476f809c743054941e97 -> Peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Couldn't send request to peer 36fc7a106457453ba91f05b78a5d342b. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:48.901109 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 27283
I20240503 08:12:48.941686 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 27413
W20240503 08:12:48.942939 27428 consensus_peers.cc:485] T 43351c2d61024ae0afc96b6cf94d1741 P 19fb518a6ede476f809c743054941e97 -> Peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Couldn't send request to peer fe8fdcdebbff4703a924f83646531115. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.3:43977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:48.943090 27428 consensus_peers.cc:485] T b8d5c3e2f8e6428aa457ba21d536cea5 P 19fb518a6ede476f809c743054941e97 -> Peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Couldn't send request to peer fe8fdcdebbff4703a924f83646531115. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.3:43977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:48.947026 27534 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:12:48.955243 27558 consensus_peers.cc:485] T a61402b1e1644ab7a0769b74e40e479b P 2de33eb4315b4547889d1e6edae2f2dd -> Peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Couldn't send request to peer fe8fdcdebbff4703a924f83646531115. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.3:43977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:49.104806 31927 raft_consensus.cc:482] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4d3aef6d5624cf4adc3859dbadccb80)
I20240503 08:12:49.104981 31927 raft_consensus.cc:504] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:49.106666 31927 leader_election.cc:288] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727)
W20240503 08:12:49.107913 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111)
W20240503 08:12:49.112541 28680 leader_election.cc:334] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111)
W20240503 08:12:49.113461 28683 leader_election.cc:334] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111)
I20240503 08:12:49.114908 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 352 } } ignore_live_leader: false dest_uuid: "b4654488aec741f299e533fdf3715ce7" is_pre_election: true
I20240503 08:12:49.115589 28681 leader_election.cc:302] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2; no voters: 36fc7a106457453ba91f05b78a5d342b, b4654488aec741f299e533fdf3715ce7, f4d3aef6d5624cf4adc3859dbadccb80
I20240503 08:12:49.115866 31927 raft_consensus.cc:2726] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:12:49.169363 31927 raft_consensus.cc:482] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4d3aef6d5624cf4adc3859dbadccb80)
I20240503 08:12:49.169523 31927 raft_consensus.cc:504] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } }
I20240503 08:12:49.170436 31927 leader_election.cc:288] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395)
W20240503 08:12:49.171764 28680 leader_election.cc:334] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111)
W20240503 08:12:49.171900 28683 leader_election.cc:334] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111)
W20240503 08:12:49.172161 28680 leader_election.cc:334] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Network error: Client connection negotiation failed: client connection to 127.24.123.3:43977: connect: Connection refused (error 111)
I20240503 08:12:49.172297 28680 leader_election.cc:302] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2; no voters: 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115
I20240503 08:12:49.172518 31927 raft_consensus.cc:2726] T c42929dcee3c461c8db91346ca1ab0f5 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:12:49.187916 31927 raft_consensus.cc:482] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4d3aef6d5624cf4adc3859dbadccb80)
I20240503 08:12:49.188066 31927 raft_consensus.cc:504] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:49.188832 31927 leader_election.cc:288] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727)
I20240503 08:12:49.189658 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 445 } } ignore_live_leader: false dest_uuid: "b4654488aec741f299e533fdf3715ce7" is_pre_election: true
W20240503 08:12:49.190021 28683 leader_election.cc:334] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111)
W20240503 08:12:49.190160 28680 leader_election.cc:334] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Network error: Client connection negotiation failed: client connection to 127.24.123.3:43977: connect: Connection refused (error 111)
I20240503 08:12:49.190287 28681 leader_election.cc:302] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2; no voters: b4654488aec741f299e533fdf3715ce7, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115
I20240503 08:12:49.190516 31927 raft_consensus.cc:2726] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:12:49.216161 27491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42929dcee3c461c8db91346ca1ab0f5" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 469 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:12:49.216667 27490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 445 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:12:49.215891 31933 raft_consensus.cc:482] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4d3aef6d5624cf4adc3859dbadccb80)
I20240503 08:12:49.217442 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 352 } } ignore_live_leader: false dest_uuid: "19fb518a6ede476f809c743054941e97" is_pre_election: true
I20240503 08:12:49.217657 27494 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 1.
I20240503 08:12:49.217180 31933 raft_consensus.cc:504] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:49.220136 27490 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1980d01e2e3e4840a8ddfca4d4e366b2 in term 1.
W20240503 08:12:49.222674 27431 leader_election.cc:334] T 8f94c738c0f34d04bd933cc09c05d6a6 P 19fb518a6ede476f809c743054941e97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111)
I20240503 08:12:49.222985 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 27413
W20240503 08:12:49.245636 27558 consensus_peers.cc:485] T 89392b248b7746d5bf8b95ac9d662e43 P 2de33eb4315b4547889d1e6edae2f2dd -> Peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Couldn't send request to peer fe8fdcdebbff4703a924f83646531115. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.3:43977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:49.271899 26950 connection.cc:535] server connection from 127.24.123.4:48097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:49.271880 30696 connection.cc:535] client connection to 127.24.123.4:41395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:49.272313 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 27543
W20240503 08:12:49.273032 30696 consensus_peers.cc:485] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 -> Peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395): Couldn't send request to peer 19fb518a6ede476f809c743054941e97. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.4:41395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:12:49.273823 28681 leader_election.cc:334] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395): Network error: Client connection negotiation failed: client connection to 127.24.123.4:41395: connect: Connection refused (error 111)
I20240503 08:12:49.278352 27664 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:49.413093 31917 raft_consensus.cc:482] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4d3aef6d5624cf4adc3859dbadccb80)
I20240503 08:12:49.413339 31917 raft_consensus.cc:504] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:49.414603 31917 leader_election.cc:288] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167)
I20240503 08:12:49.415449 28746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f94c738c0f34d04bd933cc09c05d6a6" candidate_uuid: "b4654488aec741f299e533fdf3715ce7" candidate_term: 2 candidate_status { last_received { term: 1 index: 445 } } ignore_live_leader: false dest_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" is_pre_election: true
I20240503 08:12:49.415702 28746 raft_consensus.cc:2454] T 8f94c738c0f34d04bd933cc09c05d6a6 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4654488aec741f299e533fdf3715ce7 in term 1.
W20240503 08:12:49.416649 30696 leader_election.cc:334] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395): Network error: Client connection negotiation failed: client connection to 127.24.123.4:41395: connect: Connection refused (error 111)
W20240503 08:12:49.417021 30698 leader_election.cc:334] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111)
W20240503 08:12:49.417398 30695 leader_election.cc:334] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Network error: Client connection negotiation failed: client connection to 127.24.123.3:43977: connect: Connection refused (error 111)
I20240503 08:12:49.417515 30695 leader_election.cc:302] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2, b4654488aec741f299e533fdf3715ce7; no voters: 19fb518a6ede476f809c743054941e97, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115
I20240503 08:12:49.417783 31917 raft_consensus.cc:2726] T 8f94c738c0f34d04bd933cc09c05d6a6 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:12:49.446797 31927 raft_consensus.cc:482] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36fc7a106457453ba91f05b78a5d342b)
I20240503 08:12:49.447023 31927 raft_consensus.cc:504] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:49.448158 31927 leader_election.cc:288] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), b4654488aec741f299e533fdf3715ce7 (127.24.123.8:35727)
I20240503 08:12:49.448918 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 474 } } ignore_live_leader: false dest_uuid: "b4654488aec741f299e533fdf3715ce7" is_pre_election: true
W20240503 08:12:49.449822 28680 leader_election.cc:334] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111)
I20240503 08:12:49.454558 27751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 474 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
I20240503 08:12:49.455065 28683 leader_election.cc:302] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2; no voters: 36fc7a106457453ba91f05b78a5d342b, b4654488aec741f299e533fdf3715ce7, d8d7c32cb9a241c99853a6639451c4ed
I20240503 08:12:49.455294 31927 raft_consensus.cc:2726] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:12:49.511837 31942 raft_consensus.cc:482] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36fc7a106457453ba91f05b78a5d342b)
I20240503 08:12:49.512012 31942 raft_consensus.cc:504] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } }
I20240503 08:12:49.513715 31942 leader_election.cc:288] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
W20240503 08:12:49.515307 27691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111)
W20240503 08:12:49.516472 27688 leader_election.cc:334] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111)
W20240503 08:12:49.516664 27691 leader_election.cc:334] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111)
W20240503 08:12:49.516800 27688 leader_election.cc:334] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977): Network error: Client connection negotiation failed: client connection to 127.24.123.3:43977: connect: Connection refused (error 111)
I20240503 08:12:49.516898 27688 leader_election.cc:302] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d8d7c32cb9a241c99853a6639451c4ed; no voters: 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80, fe8fdcdebbff4703a924f83646531115
I20240503 08:12:49.517217 31942 raft_consensus.cc:2726] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:12:49.552033 31949 raft_consensus.cc:482] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36fc7a106457453ba91f05b78a5d342b)
I20240503 08:12:49.552211 31949 raft_consensus.cc:504] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "fe8fdcdebbff4703a924f83646531115" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 43977 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } attrs { promote: false } }
I20240503 08:12:49.552765 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 474 } } ignore_live_leader: false dest_uuid: "2de33eb4315b4547889d1e6edae2f2dd" is_pre_election: true
I20240503 08:12:49.553112 31949 leader_election.cc:288] T 8e4a8224c7c1430a9354e58bd16c7abd P 2de33eb4315b4547889d1e6edae2f2dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), fe8fdcdebbff4703a924f83646531115 (127.24.123.3:43977), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261)
I20240503 08:12:49.556489 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 27543
W20240503 08:12:49.595963 30695 connection.cc:535] server connection from 127.24.123.5:49681 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:49.596073 27689 connection.cc:535] client connection to 127.24.123.5:43185 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:12:49.596271 26950 connection.cc:535] server connection from 127.24.123.5:35571 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:12:49.596591 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 27673
I20240503 08:12:49.596565 31917 raft_consensus.cc:482] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36fc7a106457453ba91f05b78a5d342b)
I20240503 08:12:49.596746 31917 raft_consensus.cc:504] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
W20240503 08:12:49.597359 27689 leader_election.cc:334] T 8e4a8224c7c1430a9354e58bd16c7abd P d8d7c32cb9a241c99853a6639451c4ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185): Network error: Client connection negotiation failed: client connection to 127.24.123.5:43185: connect: Connection refused (error 111)
I20240503 08:12:49.597934 31917 leader_election.cc:288] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465), 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185), 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167)
I20240503 08:12:49.598462 28750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "b4654488aec741f299e533fdf3715ce7" candidate_term: 2 candidate_status { last_received { term: 1 index: 474 } } ignore_live_leader: false dest_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" is_pre_election: true
I20240503 08:12:49.598661 28750 raft_consensus.cc:2454] T 4c364e9ddad74e8e882df9080d9d5372 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4654488aec741f299e533fdf3715ce7 in term 1.
I20240503 08:12:49.598894 27751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c364e9ddad74e8e882df9080d9d5372" candidate_uuid: "b4654488aec741f299e533fdf3715ce7" candidate_term: 2 candidate_status { last_received { term: 1 index: 474 } } ignore_live_leader: false dest_uuid: "d8d7c32cb9a241c99853a6639451c4ed" is_pre_election: true
W20240503 08:12:49.599650 30696 leader_election.cc:334] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2de33eb4315b4547889d1e6edae2f2dd (127.24.123.5:43185): Network error: Client connection negotiation failed: client connection to 127.24.123.5:43185: connect: Connection refused (error 111)
W20240503 08:12:49.599851 30695 leader_election.cc:334] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111)
I20240503 08:12:49.599965 30695 leader_election.cc:302] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2, b4654488aec741f299e533fdf3715ce7; no voters: 2de33eb4315b4547889d1e6edae2f2dd, 36fc7a106457453ba91f05b78a5d342b, d8d7c32cb9a241c99853a6639451c4ed
I20240503 08:12:49.600198 31917 raft_consensus.cc:2726] T 4c364e9ddad74e8e882df9080d9d5372 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:12:49.602167 27795 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:49.735142 31917 raft_consensus.cc:482] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4d3aef6d5624cf4adc3859dbadccb80)
I20240503 08:12:49.735359 31917 raft_consensus.cc:504] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "19fb518a6ede476f809c743054941e97" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 41395 } } peers { permanent_uuid: "f4d3aef6d5624cf4adc3859dbadccb80" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39261 } } peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:49.736599 31917 leader_election.cc:288] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395), f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261), 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503), 1980d01e2e3e4840a8ddfca4d4e366b2 (127.24.123.7:41167)
I20240503 08:12:49.737475 28750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c93cddbbf84976baa817ab0d98a811" candidate_uuid: "b4654488aec741f299e533fdf3715ce7" candidate_term: 2 candidate_status { last_received { term: 1 index: 352 } } ignore_live_leader: false dest_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" is_pre_election: true
I20240503 08:12:49.737706 28750 raft_consensus.cc:2454] T e8c93cddbbf84976baa817ab0d98a811 P 1980d01e2e3e4840a8ddfca4d4e366b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4654488aec741f299e533fdf3715ce7 in term 1.
W20240503 08:12:49.737737 30696 leader_election.cc:334] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 19fb518a6ede476f809c743054941e97 (127.24.123.4:41395): Network error: Client connection negotiation failed: client connection to 127.24.123.4:41395: connect: Connection refused (error 111)
W20240503 08:12:49.738049 30695 leader_election.cc:334] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36fc7a106457453ba91f05b78a5d342b (127.24.123.2:33503): Network error: Client connection negotiation failed: client connection to 127.24.123.2:33503: connect: Connection refused (error 111)
W20240503 08:12:49.738394 30698 leader_election.cc:334] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f4d3aef6d5624cf4adc3859dbadccb80 (127.24.123.1:39261): Network error: Client connection negotiation failed: client connection to 127.24.123.1:39261: connect: Connection refused (error 111)
I20240503 08:12:49.738512 30698 leader_election.cc:302] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 1980d01e2e3e4840a8ddfca4d4e366b2, b4654488aec741f299e533fdf3715ce7; no voters: 19fb518a6ede476f809c743054941e97, 36fc7a106457453ba91f05b78a5d342b, f4d3aef6d5624cf4adc3859dbadccb80
I20240503 08:12:49.738744 31917 raft_consensus.cc:2726] T e8c93cddbbf84976baa817ab0d98a811 P b4654488aec741f299e533fdf3715ce7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:12:49.847180 31942 raft_consensus.cc:482] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36fc7a106457453ba91f05b78a5d342b)
I20240503 08:12:49.847378 31942 raft_consensus.cc:504] T 4c364e9ddad74e8e882df9080d9d5372 P d8d7c32cb9a241c99853a6639451c4ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 474 OBSOLETE_local: false peers { permanent_uuid: "36fc7a106457453ba91f05b78a5d342b" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33503 } } peers { permanent_uuid: "d8d7c32cb9a241c99853a6639451c4ed" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 43465 } } peers { permanent_uuid: "2de33eb4315b4547889d1e6edae2f2dd" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43185 } } peers { permanent_uuid: "1980d01e2e3e4840a8ddfca4d4e366b2" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 41167 } attrs { promote: false } } peers { permanent_uuid: "b4654488aec741f299e533fdf3715ce7" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 35727 } attrs { promote: false } }
I20240503 08:12:49.847739 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 27673
I20240503 08:12:49.888943 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 28663
I20240503 08:12:49.894408 28791 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:12:50.020769 30698 consensus_peers.cc:485] T 921b4706d2bc4693ae5a0a10b310d5d4 P b4654488aec741f299e533fdf3715ce7 -> Peer d8d7c32cb9a241c99853a6639451c4ed (127.24.123.6:43465): Couldn't send request to peer d8d7c32cb9a241c99853a6639451c4ed. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.6:43465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:50.125986 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 28663
I20240503 08:12:50.163223 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 30678
I20240503 08:12:50.168594 30802 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:50.332597 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 30678
I20240503 08:12:50.354662 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 26932
I20240503 08:12:50.355728 26992 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:50.533735 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 26932
2024-05-03T08:12:50Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/7 (89113 ms)
[----------] 1 test from RebalanceParamTest (89113 ms total)
[----------] 1 test from TserverAddedDuringRebalancingTest
[ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1
I20240503 08:12:50.656912 25068 test_util.cc:268] Using random seed: 1227401844
I20240503 08:12:50.658505 25068 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:12:50.658562 25068 ts_itest-base.cc:113] --------------
I20240503 08:12:50.658594 25068 ts_itest-base.cc:114] 8 tablet servers
I20240503 08:12:50.658641 25068 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:12:50.658689 25068 ts_itest-base.cc:116] --------------
2024-05-03T08:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:12:50Z Disabled control of system clock
I20240503 08:12:50.699983 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:44195
--webserver_interface=127.24.123.62
--webserver_port=0
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.123.62:44195
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:12:51.015134 31969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:51.015408 31969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:51.015496 31969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:51.022532 31969 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:51.022657 31969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:12:51.022713 31969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:51.022768 31969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:12:51.022811 31969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:12:51.031409 31969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.123.62:44195
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:44195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.123.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:51.032584 31969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:51.034202 31969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:51.044190 31974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:51.044530 31977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:51.045063 31975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:51.045542 31969 server_base.cc:1034] running on GCE node
I20240503 08:12:51.046192 31969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:51.047488 31969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:51.048698 31969 hybrid_clock.cc:648] HybridClock initialized: now 1714723971048648 us; error 36 us; skew 500 ppm
I20240503 08:12:51.049027 31969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:51.051070 31969 webserver.cc:460] Webserver started at http://127.24.123.62:39385/ using document root <none> and password file <none>
I20240503 08:12:51.051755 31969 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:51.051862 31969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:51.052157 31969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:51.054556 31969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c96f604a3b3640f2b9f75fc4380c08d4"
format_stamp: "Formatted at 2024-05-03 08:12:51 on dist-test-slave-gkrw"
I20240503 08:12:51.055275 31969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c96f604a3b3640f2b9f75fc4380c08d4"
format_stamp: "Formatted at 2024-05-03 08:12:51 on dist-test-slave-gkrw"
I20240503 08:12:51.059975 31969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:51.062973 31983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:51.064182 31969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:51.064332 31969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c96f604a3b3640f2b9f75fc4380c08d4"
format_stamp: "Formatted at 2024-05-03 08:12:51 on dist-test-slave-gkrw"
I20240503 08:12:51.064503 31969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:51.085116 31969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:51.085909 31969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:51.086169 31969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:51.102378 31969 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.62:44195
I20240503 08:12:51.102398 32034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.62:44195 every 8 connection(s)
I20240503 08:12:51.103623 31969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:12:51.104085 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 31969
I20240503 08:12:51.104269 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:12:51.107506 32035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.117929 32035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4: Bootstrap starting.
I20240503 08:12:51.120977 32035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.121894 32035 log.cc:824] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:51.124119 32035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4: No bootstrap required, opened a new log
I20240503 08:12:51.129907 32035 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96f604a3b3640f2b9f75fc4380c08d4" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44195 } }
I20240503 08:12:51.130194 32035 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.130266 32035 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96f604a3b3640f2b9f75fc4380c08d4, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.130743 32035 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96f604a3b3640f2b9f75fc4380c08d4" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44195 } }
I20240503 08:12:51.130947 32035 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:51.131052 32035 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:51.131162 32035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.133339 32035 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96f604a3b3640f2b9f75fc4380c08d4" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44195 } }
I20240503 08:12:51.133739 32035 leader_election.cc:302] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96f604a3b3640f2b9f75fc4380c08d4; no voters:
I20240503 08:12:51.134222 32035 leader_election.cc:288] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:51.134367 32040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.135326 32040 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [term 1 LEADER]: Becoming Leader. State: Replica: c96f604a3b3640f2b9f75fc4380c08d4, State: Running, Role: LEADER
I20240503 08:12:51.135771 32040 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96f604a3b3640f2b9f75fc4380c08d4" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44195 } }
I20240503 08:12:51.135936 32035 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:51.138391 32042 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96f604a3b3640f2b9f75fc4380c08d4. Latest consensus state: current_term: 1 leader_uuid: "c96f604a3b3640f2b9f75fc4380c08d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96f604a3b3640f2b9f75fc4380c08d4" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44195 } } }
I20240503 08:12:51.138684 32042 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:51.139125 32041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96f604a3b3640f2b9f75fc4380c08d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96f604a3b3640f2b9f75fc4380c08d4" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44195 } } }
I20240503 08:12:51.139586 32041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:51.144193 32046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:51.149314 32046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:51.157491 32046 catalog_manager.cc:1347] Generated new cluster ID: c00c3ebd0d544d4a848bb7e6ff2d2cd6
I20240503 08:12:51.157616 32046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:51.186918 32046 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:51.188496 32046 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:51.195745 32046 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c96f604a3b3640f2b9f75fc4380c08d4: Generated new TSK 0
I20240503 08:12:51.196554 32046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:51.203487 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.1:0
--local_ip_for_outbound_sockets=127.24.123.1
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44195
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 08:12:51.523007 32059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:51.523273 32059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:51.523401 32059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:51.530541 32059 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:51.530644 32059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:51.530797 32059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.1
I20240503 08:12:51.539377 32059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.123.1
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.1
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:51.540534 32059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:51.542105 32059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:51.553483 32064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:51.554028 32065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:51.554448 32067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:51.555058 32059 server_base.cc:1034] running on GCE node
I20240503 08:12:51.555580 32059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:51.556424 32059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:51.557605 32059 hybrid_clock.cc:648] HybridClock initialized: now 1714723971557562 us; error 40 us; skew 500 ppm
I20240503 08:12:51.557883 32059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:51.563585 32059 webserver.cc:460] Webserver started at http://127.24.123.1:35315/ using document root <none> and password file <none>
I20240503 08:12:51.564270 32059 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:51.564378 32059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:51.564682 32059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:51.569083 32059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
format_stamp: "Formatted at 2024-05-03 08:12:51 on dist-test-slave-gkrw"
I20240503 08:12:51.569799 32059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
format_stamp: "Formatted at 2024-05-03 08:12:51 on dist-test-slave-gkrw"
I20240503 08:12:51.574707 32059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240503 08:12:51.577893 32073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:51.579174 32059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:51.579314 32059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
format_stamp: "Formatted at 2024-05-03 08:12:51 on dist-test-slave-gkrw"
I20240503 08:12:51.579483 32059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:51.597970 32059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:51.598714 32059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:51.599004 32059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:51.599856 32059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:51.601287 32059 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:51.601385 32059 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:51.601480 32059 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:51.601552 32059 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:51.628829 32059 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.1:41731
I20240503 08:12:51.628883 32185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.1:41731 every 8 connection(s)
I20240503 08:12:51.630055 32059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:12:51.631179 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 32059
I20240503 08:12:51.631397 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:12:51.634721 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.2:0
--local_ip_for_outbound_sockets=127.24.123.2
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44195
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:12:51.642225 32186 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:12:51.642483 32186 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:51.643394 32186 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:12:51.645462 32000 ts_manager.cc:194] Registered new tserver with Master: 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:12:51.646862 32000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.1:56253
W20240503 08:12:51.960713 32190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:51.960978 32190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:51.961131 32190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:51.968353 32190 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:51.968497 32190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:51.968673 32190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.2
I20240503 08:12:51.977496 32190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.123.2
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.2
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:51.978730 32190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:51.980422 32190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:51.991869 32195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:51.992262 32198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:51.992764 32196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:51.993345 32190 server_base.cc:1034] running on GCE node
I20240503 08:12:51.993870 32190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:51.994721 32190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:51.995944 32190 hybrid_clock.cc:648] HybridClock initialized: now 1714723971995888 us; error 38 us; skew 500 ppm
I20240503 08:12:51.996327 32190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:52.002976 32190 webserver.cc:460] Webserver started at http://127.24.123.2:41573/ using document root <none> and password file <none>
I20240503 08:12:52.003686 32190 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:52.003787 32190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:52.004101 32190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:52.006572 32190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "272c59640332489181cbf2cc52d266bb"
format_stamp: "Formatted at 2024-05-03 08:12:51 on dist-test-slave-gkrw"
I20240503 08:12:52.007315 32190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "272c59640332489181cbf2cc52d266bb"
format_stamp: "Formatted at 2024-05-03 08:12:51 on dist-test-slave-gkrw"
I20240503 08:12:52.012099 32190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:12:52.015321 32204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:52.016569 32190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 08:12:52.016716 32190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "272c59640332489181cbf2cc52d266bb"
format_stamp: "Formatted at 2024-05-03 08:12:51 on dist-test-slave-gkrw"
I20240503 08:12:52.016897 32190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:52.044302 32190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:52.045186 32190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:52.045454 32190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:52.046365 32190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:52.047789 32190 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:52.047879 32190 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:52.047998 32190 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:52.048065 32190 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:52.076690 32190 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.2:39617
I20240503 08:12:52.076754 32316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.2:39617 every 8 connection(s)
I20240503 08:12:52.077970 32190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:12:52.080724 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 32190
I20240503 08:12:52.080879 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:12:52.083895 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.3:0
--local_ip_for_outbound_sockets=127.24.123.3
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44195
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:12:52.090395 32317 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:12:52.090708 32317 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:52.091632 32317 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:12:52.093102 32000 ts_manager.cc:194] Registered new tserver with Master: 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:12:52.094075 32000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.2:54095
W20240503 08:12:52.404027 32320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:52.404305 32320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:52.404412 32320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:52.411567 32320 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:52.411675 32320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:52.411818 32320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.3
I20240503 08:12:52.420534 32320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.123.3
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.3
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:52.421751 32320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:52.423442 32320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:52.434659 32325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:52.435168 32328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:52.435575 32326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:52.436136 32320 server_base.cc:1034] running on GCE node
I20240503 08:12:52.436691 32320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:52.437556 32320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:52.438730 32320 hybrid_clock.cc:648] HybridClock initialized: now 1714723972438679 us; error 45 us; skew 500 ppm
I20240503 08:12:52.439049 32320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:52.441067 32320 webserver.cc:460] Webserver started at http://127.24.123.3:35361/ using document root <none> and password file <none>
I20240503 08:12:52.441715 32320 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:52.441809 32320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:52.442106 32320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:52.444514 32320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "71ad6bf07682497aaa1e9429320aef79"
format_stamp: "Formatted at 2024-05-03 08:12:52 on dist-test-slave-gkrw"
I20240503 08:12:52.445242 32320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "71ad6bf07682497aaa1e9429320aef79"
format_stamp: "Formatted at 2024-05-03 08:12:52 on dist-test-slave-gkrw"
I20240503 08:12:52.449815 32320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 08:12:52.452806 32334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:52.453984 32320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.454128 32320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "71ad6bf07682497aaa1e9429320aef79"
format_stamp: "Formatted at 2024-05-03 08:12:52 on dist-test-slave-gkrw"
I20240503 08:12:52.454301 32320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:52.479607 32320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:52.480376 32320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:52.480619 32320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:52.481487 32320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:52.482848 32320 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:52.482964 32320 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:52.483074 32320 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:52.483139 32320 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:52.511086 32320 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.3:34237
I20240503 08:12:52.511163 32446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.3:34237 every 8 connection(s)
I20240503 08:12:52.512353 32320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:12:52.521201 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 32320
I20240503 08:12:52.521682 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:12:52.523854 32447 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:12:52.524140 32447 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:52.525040 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.4:0
--local_ip_for_outbound_sockets=127.24.123.4
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44195
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:12:52.525082 32447 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:12:52.526448 32000 ts_manager.cc:194] Registered new tserver with Master: 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:12:52.527422 32000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.3:46137
I20240503 08:12:52.649857 32186 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
W20240503 08:12:52.848917 32450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:52.849169 32450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:52.849270 32450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:52.856717 32450 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:52.856817 32450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:52.856937 32450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.4
I20240503 08:12:52.865545 32450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.123.4
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.4
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:52.866662 32450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:52.868291 32450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:52.879277 32455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:52.879667 32458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:52.880213 32456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:52.880674 32450 server_base.cc:1034] running on GCE node
I20240503 08:12:52.881207 32450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:52.882054 32450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:52.883244 32450 hybrid_clock.cc:648] HybridClock initialized: now 1714723972883198 us; error 34 us; skew 500 ppm
I20240503 08:12:52.883550 32450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:52.885628 32450 webserver.cc:460] Webserver started at http://127.24.123.4:37081/ using document root <none> and password file <none>
I20240503 08:12:52.886307 32450 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:52.886412 32450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:52.886698 32450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:52.889098 32450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "09a050703a1a4c0a8dfc04e4c2853797"
format_stamp: "Formatted at 2024-05-03 08:12:52 on dist-test-slave-gkrw"
I20240503 08:12:52.889773 32450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "09a050703a1a4c0a8dfc04e4c2853797"
format_stamp: "Formatted at 2024-05-03 08:12:52 on dist-test-slave-gkrw"
I20240503 08:12:52.894358 32450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20240503 08:12:52.897392 32464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:52.898522 32450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:52.898660 32450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "09a050703a1a4c0a8dfc04e4c2853797"
format_stamp: "Formatted at 2024-05-03 08:12:52 on dist-test-slave-gkrw"
I20240503 08:12:52.898828 32450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:52.913416 32450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:52.914124 32450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:52.914359 32450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:52.915205 32450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:52.916522 32450 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:52.916616 32450 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:52.916711 32450 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:52.916776 32450 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:52.943723 32450 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.4:46785
I20240503 08:12:52.943799 32576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.4:46785 every 8 connection(s)
I20240503 08:12:52.944972 32450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:12:52.951203 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 32450
I20240503 08:12:52.951387 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:12:52.954406 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.5:0
--local_ip_for_outbound_sockets=127.24.123.5
--webserver_interface=127.24.123.5
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44195
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:12:52.956609 32577 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:12:52.956884 32577 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:52.957748 32577 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:12:52.966049 32000 ts_manager.cc:194] Registered new tserver with Master: 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:12:52.966961 32000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.4:36405
I20240503 08:12:53.096447 32317 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
W20240503 08:12:53.267864 32580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:53.268147 32580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:53.268261 32580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:53.275260 32580 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:53.275365 32580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:53.275544 32580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.5
I20240503 08:12:53.284112 32580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.123.5
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.5
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:53.285236 32580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:53.286789 32580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:53.298190 32588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:53.298630 32585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:53.299033 32586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:53.299614 32580 server_base.cc:1034] running on GCE node
I20240503 08:12:53.300123 32580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:53.300961 32580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:53.302165 32580 hybrid_clock.cc:648] HybridClock initialized: now 1714723973302124 us; error 42 us; skew 500 ppm
I20240503 08:12:53.302464 32580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:53.304529 32580 webserver.cc:460] Webserver started at http://127.24.123.5:45317/ using document root <none> and password file <none>
I20240503 08:12:53.305171 32580 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:53.305271 32580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:53.305555 32580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:53.307904 32580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c6b280e8ecd54855b7b21b58a223149a"
format_stamp: "Formatted at 2024-05-03 08:12:53 on dist-test-slave-gkrw"
I20240503 08:12:53.308593 32580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c6b280e8ecd54855b7b21b58a223149a"
format_stamp: "Formatted at 2024-05-03 08:12:53 on dist-test-slave-gkrw"
I20240503 08:12:53.313133 32580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:12:53.316033 32594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:53.317144 32580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:53.317279 32580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c6b280e8ecd54855b7b21b58a223149a"
format_stamp: "Formatted at 2024-05-03 08:12:53 on dist-test-slave-gkrw"
I20240503 08:12:53.317451 32580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:53.346033 32580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:53.346717 32580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:53.346973 32580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:53.347779 32580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:53.349081 32580 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:53.349169 32580 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:53.349267 32580 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:53.349334 32580 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:53.376238 32580 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.5:44035
I20240503 08:12:53.376298 32706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.5:44035 every 8 connection(s)
I20240503 08:12:53.377384 32580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:12:53.382624 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 32580
I20240503 08:12:53.382769 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:12:53.385774 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.6:0
--local_ip_for_outbound_sockets=127.24.123.6
--webserver_interface=127.24.123.6
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44195
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:12:53.388942 32707 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:12:53.389179 32707 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:53.389886 32707 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:12:53.391132 32000 ts_manager.cc:194] Registered new tserver with Master: c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035)
I20240503 08:12:53.391844 32000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.5:43245
I20240503 08:12:53.531180 32447 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
W20240503 08:12:53.703621 32710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:53.703882 32710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:53.703986 32710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:53.711105 32710 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:53.711202 32710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:53.711328 32710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.6
I20240503 08:12:53.719949 32710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.123.6
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.6
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:53.721200 32710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:53.722857 32710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:53.734326 32715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:53.734771 32718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:53.735186 32716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:53.735743 32710 server_base.cc:1034] running on GCE node
I20240503 08:12:53.736311 32710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:53.737167 32710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:53.738350 32710 hybrid_clock.cc:648] HybridClock initialized: now 1714723973738306 us; error 40 us; skew 500 ppm
I20240503 08:12:53.738618 32710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:53.740681 32710 webserver.cc:460] Webserver started at http://127.24.123.6:32843/ using document root <none> and password file <none>
I20240503 08:12:53.741344 32710 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:53.741436 32710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:53.741688 32710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:53.744210 32710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6c93e47973e047d0a162a50f72f9a757"
format_stamp: "Formatted at 2024-05-03 08:12:53 on dist-test-slave-gkrw"
I20240503 08:12:53.744936 32710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6c93e47973e047d0a162a50f72f9a757"
format_stamp: "Formatted at 2024-05-03 08:12:53 on dist-test-slave-gkrw"
I20240503 08:12:53.749732 32710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 08:12:53.752816 32724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:53.753991 32710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:53.754153 32710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c93e47973e047d0a162a50f72f9a757"
format_stamp: "Formatted at 2024-05-03 08:12:53 on dist-test-slave-gkrw"
I20240503 08:12:53.754321 32710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:53.775516 32710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:53.776296 32710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:53.776546 32710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:53.777432 32710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:53.778824 32710 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:53.778939 32710 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:53.779042 32710 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:53.779107 32710 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:53.811015 32710 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.6:36447
I20240503 08:12:53.811120 368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.6:36447 every 8 connection(s)
I20240503 08:12:53.812295 32710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:12:53.821578 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 32710
I20240503 08:12:53.821780 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:12:53.824151 369 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:12:53.824456 369 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:53.825076 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.7:0
--local_ip_for_outbound_sockets=127.24.123.7
--webserver_interface=127.24.123.7
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44195
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:12:53.825387 369 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:12:53.826689 32000 ts_manager.cc:194] Registered new tserver with Master: 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447)
I20240503 08:12:53.827673 32000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.6:34147
I20240503 08:12:53.969345 32577 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
W20240503 08:12:54.140065 372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:54.140319 372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:54.140450 372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:54.147634 372 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:54.147733 372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:54.147881 372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.7
I20240503 08:12:54.156400 372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.123.7
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.7
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:54.157596 372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:54.159344 372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:54.170513 380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:54.170825 378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:54.171202 377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:54.171818 372 server_base.cc:1034] running on GCE node
I20240503 08:12:54.172376 372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:54.173218 372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:54.174412 372 hybrid_clock.cc:648] HybridClock initialized: now 1714723974174370 us; error 31 us; skew 500 ppm
I20240503 08:12:54.174711 372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:54.176781 372 webserver.cc:460] Webserver started at http://127.24.123.7:33787/ using document root <none> and password file <none>
I20240503 08:12:54.177448 372 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:54.177548 372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:54.177842 372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:54.180305 372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f7f6592d13d245508261964ca6ffc5c4"
format_stamp: "Formatted at 2024-05-03 08:12:54 on dist-test-slave-gkrw"
I20240503 08:12:54.181056 372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f7f6592d13d245508261964ca6ffc5c4"
format_stamp: "Formatted at 2024-05-03 08:12:54 on dist-test-slave-gkrw"
I20240503 08:12:54.185861 372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 08:12:54.188906 386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:54.190104 372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:54.190246 372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f7f6592d13d245508261964ca6ffc5c4"
format_stamp: "Formatted at 2024-05-03 08:12:54 on dist-test-slave-gkrw"
I20240503 08:12:54.190426 372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:54.208245 372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:54.208995 372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:54.209236 372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:54.210081 372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:54.211434 372 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:54.211525 372 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:54.211628 372 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:54.211695 372 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:54.239591 372 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.7:33603
I20240503 08:12:54.239642 498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.7:33603 every 8 connection(s)
I20240503 08:12:54.240810 372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:12:54.250617 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 372
I20240503 08:12:54.250809 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:12:54.252665 499 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:12:54.252928 499 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:54.253733 499 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:12:54.254004 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.8:0
--local_ip_for_outbound_sockets=127.24.123.8
--webserver_interface=127.24.123.8
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44195
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:12:54.255056 31999 ts_manager.cc:194] Registered new tserver with Master: f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603)
I20240503 08:12:54.255858 31999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.7:58265
I20240503 08:12:54.394042 32707 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
W20240503 08:12:54.567602 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:54.567848 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:54.567950 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:54.575071 502 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:54.575176 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:54.575299 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.8
I20240503 08:12:54.584034 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.123.8
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.8
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:54.585208 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:54.586840 502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:54.598285 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:54.598603 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:54.599210 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:54.599640 502 server_base.cc:1034] running on GCE node
I20240503 08:12:54.600230 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:54.601155 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:54.602385 502 hybrid_clock.cc:648] HybridClock initialized: now 1714723974602325 us; error 46 us; skew 500 ppm
I20240503 08:12:54.602687 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:54.604902 502 webserver.cc:460] Webserver started at http://127.24.123.8:35997/ using document root <none> and password file <none>
I20240503 08:12:54.605599 502 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:54.605715 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:54.606052 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:54.608702 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e615bd1588964e2fbe14f14f4aae0a98"
format_stamp: "Formatted at 2024-05-03 08:12:54 on dist-test-slave-gkrw"
I20240503 08:12:54.609480 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e615bd1588964e2fbe14f14f4aae0a98"
format_stamp: "Formatted at 2024-05-03 08:12:54 on dist-test-slave-gkrw"
I20240503 08:12:54.614434 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 08:12:54.617681 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:54.618916 502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:54.619117 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e615bd1588964e2fbe14f14f4aae0a98"
format_stamp: "Formatted at 2024-05-03 08:12:54 on dist-test-slave-gkrw"
I20240503 08:12:54.619321 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:54.654287 502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:54.655251 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:54.655580 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:54.656522 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:54.657999 502 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:54.658119 502 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:54.658226 502 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:54.658296 502 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:54.686807 502 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.8:43749
I20240503 08:12:54.686893 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.8:43749 every 8 connection(s)
I20240503 08:12:54.688069 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:12:54.689580 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 502
I20240503 08:12:54.689738 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:12:54.699880 629 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:12:54.700187 629 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:54.700904 629 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:12:54.702121 31999 ts_manager.cc:194] Registered new tserver with Master: e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749)
I20240503 08:12:54.702900 31999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.8:54489
I20240503 08:12:54.705469 25068 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:12:54.720242 25068 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:12:54.720364 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 32580
I20240503 08:12:54.727769 32702 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:54.807504 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 32580
I20240503 08:12:54.823097 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 32710
I20240503 08:12:54.830184 369 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
I20240503 08:12:54.830562 364 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:54.908775 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 32710
I20240503 08:12:54.921169 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 372
I20240503 08:12:54.928630 494 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:55.006328 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 372
I20240503 08:12:55.018945 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 502
I20240503 08:12:55.026829 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:12:55.104882 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 502
I20240503 08:12:58.871570 31999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:58.906132 32382 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a81a3850e49edb08c8060a45b6831 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.907625 32381 tablet_service.cc:1467] Processing CreateTablet for tablet 27a469ffc17447ed86860e991ff3534a (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.907967 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a81a3850e49edb08c8060a45b6831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.908838 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a469ffc17447ed86860e991ff3534a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.913792 32252 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a81a3850e49edb08c8060a45b6831 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.913794 32512 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a81a3850e49edb08c8060a45b6831 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.913806 32120 tablet_service.cc:1467] Processing CreateTablet for tablet 25f1956062e84b92a3b036fc5dff4b46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.915233 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a81a3850e49edb08c8060a45b6831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.915287 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f1956062e84b92a3b036fc5dff4b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.914911 32250 tablet_service.cc:1467] Processing CreateTablet for tablet 25f1956062e84b92a3b036fc5dff4b46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.915172 32251 tablet_service.cc:1467] Processing CreateTablet for tablet 27a469ffc17447ed86860e991ff3534a (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.916108 32250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f1956062e84b92a3b036fc5dff4b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.916481 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a469ffc17447ed86860e991ff3534a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.919344 32252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a81a3850e49edb08c8060a45b6831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.919281 32121 tablet_service.cc:1467] Processing CreateTablet for tablet 27a469ffc17447ed86860e991ff3534a (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.919735 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a469ffc17447ed86860e991ff3534a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.920544 32511 tablet_service.cc:1467] Processing CreateTablet for tablet 25f1956062e84b92a3b036fc5dff4b46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.921008 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f1956062e84b92a3b036fc5dff4b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.942963 660 tablet_bootstrap.cc:492] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:12:58.946031 661 tablet_bootstrap.cc:492] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:12:58.947278 660 tablet_bootstrap.cc:654] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.948539 660 log.cc:824] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:58.949566 661 tablet_bootstrap.cc:654] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.951520 658 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:12:58.952217 661 log.cc:824] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:58.955221 658 tablet_bootstrap.cc:654] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.956036 659 tablet_bootstrap.cc:492] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:12:58.957222 658 log.cc:824] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:58.957917 660 tablet_bootstrap.cc:492] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:12:58.958281 660 ts_tablet_manager.cc:1367] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240503 08:12:58.959790 659 tablet_bootstrap.cc:654] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.960989 659 log.cc:824] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:58.961213 661 tablet_bootstrap.cc:492] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:12:58.961515 661 ts_tablet_manager.cc:1367] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240503 08:12:58.968739 660 raft_consensus.cc:348] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:58.969336 660 raft_consensus.cc:374] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.969480 660 raft_consensus.cc:729] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.970000 659 tablet_bootstrap.cc:492] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:12:58.970161 660 consensus_queue.cc:260] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:58.970330 659 ts_tablet_manager.cc:1367] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20240503 08:12:58.971004 661 raft_consensus.cc:348] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:58.971498 661 raft_consensus.cc:374] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.971639 661 raft_consensus.cc:729] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.971714 658 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:12:58.972064 658 ts_tablet_manager.cc:1367] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.014s
I20240503 08:12:58.972499 661 consensus_queue.cc:260] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:58.978691 659 raft_consensus.cc:348] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:58.979211 659 raft_consensus.cc:374] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.979343 659 raft_consensus.cc:729] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.980377 658 raft_consensus.cc:348] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:12:58.980799 658 raft_consensus.cc:374] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.980950 658 raft_consensus.cc:729] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.981717 658 consensus_queue.cc:260] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:12:58.981717 659 consensus_queue.cc:260] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:58.984676 660 ts_tablet_manager.cc:1398] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.026s user 0.012s sys 0.011s
I20240503 08:12:58.984896 659 ts_tablet_manager.cc:1398] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20240503 08:12:58.985116 660 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:12:58.986778 660 tablet_bootstrap.cc:654] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.988023 658 ts_tablet_manager.cc:1398] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.016s user 0.004s sys 0.009s
I20240503 08:12:58.988461 658 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:12:58.990180 658 tablet_bootstrap.cc:654] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.990918 661 ts_tablet_manager.cc:1398] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.029s user 0.010s sys 0.009s
I20240503 08:12:58.994252 659 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:12:58.994978 661 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:12:58.996026 659 tablet_bootstrap.cc:654] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.996675 661 tablet_bootstrap.cc:654] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.998231 658 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:12:58.998440 658 ts_tablet_manager.cc:1367] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240503 08:12:58.999533 658 raft_consensus.cc:348] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:58.999835 658 raft_consensus.cc:374] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.999967 658 raft_consensus.cc:729] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.000284 658 consensus_queue.cc:260] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.000926 658 ts_tablet_manager.cc:1398] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:59.002441 659 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:12:59.002619 659 ts_tablet_manager.cc:1367] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:12:59.002756 660 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:12:59.002784 661 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:12:59.002941 661 ts_tablet_manager.cc:1367] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:12:59.002967 660 ts_tablet_manager.cc:1367] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20240503 08:12:59.003700 659 raft_consensus.cc:348] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:12:59.003957 659 raft_consensus.cc:374] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.003904 660 raft_consensus.cc:348] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:12:59.004004 661 raft_consensus.cc:348] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.004146 660 raft_consensus.cc:374] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.004091 659 raft_consensus.cc:729] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.004215 661 raft_consensus.cc:374] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.004251 660 raft_consensus.cc:729] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.004321 661 raft_consensus.cc:729] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.004473 660 consensus_queue.cc:260] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:12:59.004561 661 consensus_queue.cc:260] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.004438 659 consensus_queue.cc:260] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:12:59.005045 661 ts_tablet_manager.cc:1398] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:59.005590 660 ts_tablet_manager.cc:1398] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:59.006206 659 ts_tablet_manager.cc:1398] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:59.006495 659 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:12:59.008023 659 tablet_bootstrap.cc:654] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.009927 659 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:12:59.010076 659 ts_tablet_manager.cc:1367] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:12:59.010571 659 raft_consensus.cc:348] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.010731 659 raft_consensus.cc:374] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.010794 659 raft_consensus.cc:729] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.011018 659 consensus_queue.cc:260] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.011505 659 ts_tablet_manager.cc:1398] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 08:12:59.017793 32448 tablet.cc:2369] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:12:59.083943 32318 tablet.cc:2369] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:12:59.136049 32187 tablet.cc:2369] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:59.162214 668 raft_consensus.cc:482] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.162442 668 raft_consensus.cc:504] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:12:59.164091 668 leader_election.cc:288] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:12:59.171001 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f1956062e84b92a3b036fc5dff4b46" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:12:59.171001 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f1956062e84b92a3b036fc5dff4b46" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:12:59.171536 32532 raft_consensus.cc:2454] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:12:59.171548 32141 raft_consensus.cc:2454] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:12:59.172238 32205 leader_election.cc:302] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:12:59.172598 668 raft_consensus.cc:2781] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.172732 668 raft_consensus.cc:482] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.172816 668 raft_consensus.cc:3037] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.174787 668 raft_consensus.cc:504] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:12:59.175387 668 leader_election.cc:288] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:12:59.175751 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f1956062e84b92a3b036fc5dff4b46" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:12:59.175951 32141 raft_consensus.cc:3037] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.175937 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f1956062e84b92a3b036fc5dff4b46" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:12:59.176118 32532 raft_consensus.cc:3037] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.178912 32532 raft_consensus.cc:2454] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:12:59.178912 32141 raft_consensus.cc:2454] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:12:59.179314 32206 leader_election.cc:302] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:12:59.179613 668 raft_consensus.cc:2781] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.180017 668 raft_consensus.cc:686] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:12:59.180464 668 consensus_queue.cc:237] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:12:59.181300 677 raft_consensus.cc:482] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.181437 677 raft_consensus.cc:504] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.182169 677 leader_election.cc:288] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:12:59.182778 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a469ffc17447ed86860e991ff3534a" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:12:59.183125 32141 raft_consensus.cc:2454] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:12:59.184253 32205 leader_election.cc:302] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:12:59.184515 668 raft_consensus.cc:2781] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.184656 668 raft_consensus.cc:482] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.184790 668 raft_consensus.cc:3037] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.185602 31998 catalog_manager.cc:5484] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:59.187230 668 raft_consensus.cc:504] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.187840 668 leader_election.cc:288] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:12:59.188033 668 raft_consensus.cc:482] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.188159 668 raft_consensus.cc:504] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.188972 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:12:59.189139 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a469ffc17447ed86860e991ff3534a" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:12:59.189324 32532 raft_consensus.cc:2454] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:12:59.189386 32141 raft_consensus.cc:3037] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.189756 32206 leader_election.cc:302] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:12:59.190027 677 raft_consensus.cc:2781] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.190133 677 raft_consensus.cc:482] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.190210 677 raft_consensus.cc:3037] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.190207 32402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a469ffc17447ed86860e991ff3534a" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:12:59.190227 32401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a469ffc17447ed86860e991ff3534a" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:12:59.190691 32401 raft_consensus.cc:3037] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.191325 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:12:59.191650 32400 raft_consensus.cc:2454] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:12:59.192170 32141 raft_consensus.cc:2454] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:12:59.192585 677 raft_consensus.cc:504] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.193284 32205 leader_election.cc:302] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:12:59.193367 32401 raft_consensus.cc:2454] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:12:59.193526 668 leader_election.cc:288] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:12:59.193662 668 raft_consensus.cc:2781] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.193799 668 raft_consensus.cc:686] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:12:59.194051 668 consensus_queue.cc:237] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:12:59.194200 677 leader_election.cc:288] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:12:59.194924 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:12:59.195168 32532 raft_consensus.cc:3037] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.195683 32401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:12:59.195843 32401 raft_consensus.cc:3037] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.197549 32401 raft_consensus.cc:2454] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:12:59.197927 32205 leader_election.cc:302] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:12:59.197934 32532 raft_consensus.cc:2454] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:12:59.198190 677 raft_consensus.cc:2781] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.198341 677 raft_consensus.cc:686] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:12:59.198559 677 consensus_queue.cc:237] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
W20240503 08:12:59.200158 32578 tablet.cc:2369] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:59.203254 31998 catalog_manager.cc:5484] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.207111 31998 catalog_manager.cc:5484] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.577351 668 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:59.581564 682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:59.581928 682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:12:59.585601 677 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.586385 677 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.593634 668 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:59.594055 682 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:12:59.704499 668 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.711041 668 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 224.006 ms
time per row: 0.112003 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:13:00.822795 32000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:00.831435 32121 tablet_service.cc:1467] Processing CreateTablet for tablet b7996d398e3440ebbb8f942bda8f6833 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.831441 32511 tablet_service.cc:1467] Processing CreateTablet for tablet b7996d398e3440ebbb8f942bda8f6833 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.831673 32381 tablet_service.cc:1467] Processing CreateTablet for tablet b7996d398e3440ebbb8f942bda8f6833 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.831892 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7996d398e3440ebbb8f942bda8f6833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.831892 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7996d398e3440ebbb8f942bda8f6833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.832124 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7996d398e3440ebbb8f942bda8f6833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.836465 32512 tablet_service.cc:1467] Processing CreateTablet for tablet 86b7e012f1544986a9460cd5dac5cece (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.836709 32251 tablet_service.cc:1467] Processing CreateTablet for tablet f9541e40695844a1b05ca1bc0dd8a10d (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.836817 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b7e012f1544986a9460cd5dac5cece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.836974 32382 tablet_service.cc:1467] Processing CreateTablet for tablet 86b7e012f1544986a9460cd5dac5cece (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.837131 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9541e40695844a1b05ca1bc0dd8a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.837301 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b7e012f1544986a9460cd5dac5cece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.839627 32120 tablet_service.cc:1467] Processing CreateTablet for tablet 86b7e012f1544986a9460cd5dac5cece (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.839987 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b7e012f1544986a9460cd5dac5cece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.843248 32380 tablet_service.cc:1467] Processing CreateTablet for tablet f9541e40695844a1b05ca1bc0dd8a10d (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.843619 32380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9541e40695844a1b05ca1bc0dd8a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.843619 32119 tablet_service.cc:1467] Processing CreateTablet for tablet f9541e40695844a1b05ca1bc0dd8a10d (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:00.844074 32119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9541e40695844a1b05ca1bc0dd8a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.853385 746 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:00.854063 744 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:00.855047 746 tablet_bootstrap.cc:654] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.855477 745 tablet_bootstrap.cc:492] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:00.855638 744 tablet_bootstrap.cc:654] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.857098 742 tablet_bootstrap.cc:492] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:00.857229 745 tablet_bootstrap.cc:654] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.859196 742 tablet_bootstrap.cc:654] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.862653 745 tablet_bootstrap.cc:492] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:00.862783 745 ts_tablet_manager.cc:1367] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:13:00.862984 744 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:00.863219 744 ts_tablet_manager.cc:1367] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 08:13:00.863526 745 raft_consensus.cc:348] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.863689 745 raft_consensus.cc:374] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.863786 745 raft_consensus.cc:729] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.863878 746 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:00.864084 746 ts_tablet_manager.cc:1367] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240503 08:13:00.863981 745 consensus_queue.cc:260] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.863996 744 raft_consensus.cc:348] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.864176 744 raft_consensus.cc:374] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.864302 744 raft_consensus.cc:729] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.864522 744 consensus_queue.cc:260] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.864764 746 raft_consensus.cc:348] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:00.865247 744 ts_tablet_manager.cc:1398] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:00.865396 746 raft_consensus.cc:374] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.865509 744 tablet_bootstrap.cc:492] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:00.865537 746 raft_consensus.cc:729] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.865831 746 consensus_queue.cc:260] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:00.867044 744 tablet_bootstrap.cc:654] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.867089 746 ts_tablet_manager.cc:1398] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:13:00.867969 742 tablet_bootstrap.cc:492] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:00.868134 742 ts_tablet_manager.cc:1367] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 08:13:00.868875 742 raft_consensus.cc:348] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.869076 742 raft_consensus.cc:374] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.869194 742 raft_consensus.cc:729] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.869436 742 consensus_queue.cc:260] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.871086 744 tablet_bootstrap.cc:492] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:00.871289 744 ts_tablet_manager.cc:1367] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 08:13:00.872036 744 raft_consensus.cc:348] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.872445 744 raft_consensus.cc:374] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.872568 744 raft_consensus.cc:729] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.872895 744 consensus_queue.cc:260] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.874936 745 ts_tablet_manager.cc:1398] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20240503 08:13:00.875253 745 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:00.877190 745 tablet_bootstrap.cc:654] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.877256 742 ts_tablet_manager.cc:1398] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240503 08:13:00.877585 742 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:00.878615 744 ts_tablet_manager.cc:1398] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20240503 08:13:00.879076 742 tablet_bootstrap.cc:654] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.880726 742 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:00.880916 742 ts_tablet_manager.cc:1367] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:13:00.882019 742 raft_consensus.cc:348] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.882160 745 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:00.882248 742 raft_consensus.cc:374] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.882303 745 ts_tablet_manager.cc:1367] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:13:00.882329 742 raft_consensus.cc:729] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.882546 742 consensus_queue.cc:260] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.883114 742 ts_tablet_manager.cc:1398] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:00.883028 745 raft_consensus.cc:348] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:00.883232 745 raft_consensus.cc:374] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.883308 745 raft_consensus.cc:729] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.883435 742 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:00.883562 745 consensus_queue.cc:260] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:00.884130 745 ts_tablet_manager.cc:1398] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:00.884423 745 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:00.884869 742 tablet_bootstrap.cc:654] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.885811 745 tablet_bootstrap.cc:654] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:00.886723 742 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:00.886943 742 ts_tablet_manager.cc:1367] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 08:13:00.887686 742 raft_consensus.cc:348] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:00.887890 742 raft_consensus.cc:374] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.887980 742 raft_consensus.cc:729] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.888208 742 consensus_queue.cc:260] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:00.888706 742 ts_tablet_manager.cc:1398] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:00.889678 745 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:00.889799 745 ts_tablet_manager.cc:1367] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:13:00.890452 745 raft_consensus.cc:348] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.890695 745 raft_consensus.cc:374] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:00.890791 745 raft_consensus.cc:729] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:00.891026 745 consensus_queue.cc:260] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.891551 745 ts_tablet_manager.cc:1398] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:00.936439 751 raft_consensus.cc:482] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:00.936622 751 raft_consensus.cc:504] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.938081 751 leader_election.cc:288] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:00.944990 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7996d398e3440ebbb8f942bda8f6833" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:00.945261 32532 raft_consensus.cc:2454] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:00.945583 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7996d398e3440ebbb8f942bda8f6833" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:00.945807 32400 raft_consensus.cc:2454] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:00.945791 32075 leader_election.cc:302] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:00.946199 751 raft_consensus.cc:2781] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:00.946319 751 raft_consensus.cc:482] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:00.946419 751 raft_consensus.cc:3037] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.949158 751 raft_consensus.cc:504] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.949818 751 leader_election.cc:288] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Requested vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:00.950207 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7996d398e3440ebbb8f942bda8f6833" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:00.950383 32532 raft_consensus.cc:3037] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.951062 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7996d398e3440ebbb8f942bda8f6833" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:00.951236 32400 raft_consensus.cc:3037] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:00.953022 32400 raft_consensus.cc:2454] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:00.953094 32532 raft_consensus.cc:2454] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:00.953439 32074 leader_election.cc:302] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:00.953717 751 raft_consensus.cc:2781] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:00.954141 751 raft_consensus.cc:686] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Leader. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:00.954607 751 consensus_queue.cc:237] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:00.958534 31995 catalog_manager.cc:5484] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:01.001873 751 raft_consensus.cc:482] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.002164 751 raft_consensus.cc:504] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.003044 751 leader_election.cc:288] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:01.003338 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:01.003427 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:01.003597 32532 raft_consensus.cc:2454] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:01.003657 32400 raft_consensus.cc:2454] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:01.003948 32075 leader_election.cc:302] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:01.004215 751 raft_consensus.cc:2781] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.004284 751 raft_consensus.cc:482] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.004360 751 raft_consensus.cc:3037] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.006076 751 raft_consensus.cc:504] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.006582 751 leader_election.cc:288] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Requested vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:01.006937 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:01.007020 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:01.007162 32532 raft_consensus.cc:3037] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.007211 32400 raft_consensus.cc:3037] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.009809 32400 raft_consensus.cc:2454] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:01.009809 32532 raft_consensus.cc:2454] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:01.010169 32074 leader_election.cc:302] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:01.010387 751 raft_consensus.cc:2781] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.010524 751 raft_consensus.cc:686] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Leader. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:01.010778 751 consensus_queue.cc:237] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.013067 31995 catalog_manager.cc:5484] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:01.083873 668 raft_consensus.cc:482] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.084086 668 raft_consensus.cc:504] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:01.084815 668 leader_election.cc:288] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:01.085259 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9541e40695844a1b05ca1bc0dd8a10d" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:01.085366 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9541e40695844a1b05ca1bc0dd8a10d" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:01.085503 32400 raft_consensus.cc:2454] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:01.085559 32141 raft_consensus.cc:2454] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:01.085906 32205 leader_election.cc:302] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:01.086158 668 raft_consensus.cc:2781] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.086284 668 raft_consensus.cc:482] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.086372 668 raft_consensus.cc:3037] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.088155 668 raft_consensus.cc:504] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:01.088609 668 leader_election.cc:288] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:01.089022 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9541e40695844a1b05ca1bc0dd8a10d" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:01.089043 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9541e40695844a1b05ca1bc0dd8a10d" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:01.089172 32400 raft_consensus.cc:3037] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.089195 32141 raft_consensus.cc:3037] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.090963 32141 raft_consensus.cc:2454] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:01.091006 32400 raft_consensus.cc:2454] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:01.091290 32205 leader_election.cc:302] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:01.091537 668 raft_consensus.cc:2781] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.091674 668 raft_consensus.cc:686] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:01.091961 668 consensus_queue.cc:237] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:01.094247 31995 catalog_manager.cc:5484] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:01.345954 751 consensus_queue.cc:1035] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.352162 764 consensus_queue.cc:1035] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.430646 764 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.435822 764 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:01.462304 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:01.462595 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:01.469790 768 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:01.481658 668 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:01.487176 668 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
W20240503 08:13:01.675468 32075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:46785, user_credentials={real_user=slave}} blocked reactor thread for 44185us
INSERT report
rows total: 2000
time total: 207.692 ms
time per row: 0.103846 ms
I20240503 08:13:01.823343 31996 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:01.831770 32251 tablet_service.cc:1467] Processing CreateTablet for tablet 2bd736780ea840ab9b6630a4acd80ae9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.832257 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd736780ea840ab9b6630a4acd80ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.835264 32120 tablet_service.cc:1467] Processing CreateTablet for tablet 2bd736780ea840ab9b6630a4acd80ae9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.835639 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd736780ea840ab9b6630a4acd80ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.836961 831 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:01.838277 831 tablet_bootstrap.cc:654] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.839756 32121 tablet_service.cc:1467] Processing CreateTablet for tablet 734d2de8d2804b2c8d2f144157e12f77 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.839762 32381 tablet_service.cc:1467] Processing CreateTablet for tablet 130b43c1e784400fb8851a71428014a1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.840118 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130b43c1e784400fb8851a71428014a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.840124 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734d2de8d2804b2c8d2f144157e12f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.845049 32511 tablet_service.cc:1467] Processing CreateTablet for tablet 2bd736780ea840ab9b6630a4acd80ae9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.845188 32252 tablet_service.cc:1467] Processing CreateTablet for tablet 734d2de8d2804b2c8d2f144157e12f77 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.845496 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd736780ea840ab9b6630a4acd80ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.845542 32252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734d2de8d2804b2c8d2f144157e12f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.847499 32251 tablet_service.cc:1467] Processing CreateTablet for tablet 130b43c1e784400fb8851a71428014a1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.847832 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130b43c1e784400fb8851a71428014a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.850670 833 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:01.851229 32512 tablet_service.cc:1467] Processing CreateTablet for tablet 130b43c1e784400fb8851a71428014a1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.851564 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130b43c1e784400fb8851a71428014a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.852423 833 tablet_bootstrap.cc:654] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.853132 831 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:01.853277 831 ts_tablet_manager.cc:1367] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 08:13:01.854060 831 raft_consensus.cc:348] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:01.854259 831 raft_consensus.cc:374] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.854338 831 raft_consensus.cc:729] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.854552 831 consensus_queue.cc:260] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:01.855356 32510 tablet_service.cc:1467] Processing CreateTablet for tablet 734d2de8d2804b2c8d2f144157e12f77 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:01.855659 831 ts_tablet_manager.cc:1398] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:01.855743 32510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734d2de8d2804b2c8d2f144157e12f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.855887 831 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:01.858304 831 tablet_bootstrap.cc:654] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.859835 834 tablet_bootstrap.cc:492] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:01.861423 834 tablet_bootstrap.cc:654] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.862809 831 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:01.862964 831 ts_tablet_manager.cc:1367] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240503 08:13:01.863559 831 raft_consensus.cc:348] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:01.863719 831 raft_consensus.cc:374] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.863788 831 raft_consensus.cc:729] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.864003 831 consensus_queue.cc:260] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:01.864557 831 ts_tablet_manager.cc:1398] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:01.864843 831 tablet_bootstrap.cc:492] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:01.866235 831 tablet_bootstrap.cc:654] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.867506 838 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:01.868880 833 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:01.869037 833 ts_tablet_manager.cc:1367] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20240503 08:13:01.869169 834 tablet_bootstrap.cc:492] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:01.869310 834 ts_tablet_manager.cc:1367] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:13:01.869797 831 tablet_bootstrap.cc:492] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:01.869925 831 ts_tablet_manager.cc:1367] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:13:01.869824 833 raft_consensus.cc:348] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:01.870003 833 raft_consensus.cc:374] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.870095 833 raft_consensus.cc:729] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.870005 834 raft_consensus.cc:348] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.870195 834 raft_consensus.cc:374] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.870281 834 raft_consensus.cc:729] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.870319 833 consensus_queue.cc:260] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:01.870499 834 consensus_queue.cc:260] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.870591 831 raft_consensus.cc:348] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.870759 831 raft_consensus.cc:374] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.870860 833 ts_tablet_manager.cc:1398] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:01.870849 831 raft_consensus.cc:729] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.871178 833 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:01.871104 831 consensus_queue.cc:260] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.871572 831 ts_tablet_manager.cc:1398] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:01.872622 833 tablet_bootstrap.cc:654] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.872905 838 tablet_bootstrap.cc:654] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.877985 834 ts_tablet_manager.cc:1398] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:13:01.879431 838 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:01.879500 833 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:01.879583 838 ts_tablet_manager.cc:1367] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240503 08:13:01.879626 833 ts_tablet_manager.cc:1367] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20240503 08:13:01.880159 833 raft_consensus.cc:348] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:01.880321 833 raft_consensus.cc:374] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.880388 833 raft_consensus.cc:729] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.880345 838 raft_consensus.cc:348] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:01.880520 838 raft_consensus.cc:374] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.880609 838 raft_consensus.cc:729] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.880602 833 consensus_queue.cc:260] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:01.881078 833 ts_tablet_manager.cc:1398] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:01.880831 838 consensus_queue.cc:260] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:01.881915 838 ts_tablet_manager.cc:1398] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:01.882206 838 tablet_bootstrap.cc:492] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:01.883517 838 tablet_bootstrap.cc:654] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.885445 838 tablet_bootstrap.cc:492] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:01.885578 838 ts_tablet_manager.cc:1367] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:01.886096 838 raft_consensus.cc:348] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.886260 838 raft_consensus.cc:374] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.886325 838 raft_consensus.cc:729] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.886535 838 consensus_queue.cc:260] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.886924 838 ts_tablet_manager.cc:1398] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:13:01.887212 838 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:01.888332 838 tablet_bootstrap.cc:654] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:01.889938 838 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:01.890069 838 ts_tablet_manager.cc:1367] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:01.890583 838 raft_consensus.cc:348] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:01.890715 838 raft_consensus.cc:374] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:01.890797 838 raft_consensus.cc:729] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:01.891017 838 consensus_queue.cc:260] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:01.891464 838 ts_tablet_manager.cc:1398] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:01.931159 743 raft_consensus.cc:482] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.931329 743 raft_consensus.cc:504] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.931957 743 leader_election.cc:288] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:01.932440 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130b43c1e784400fb8851a71428014a1" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:01.932461 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130b43c1e784400fb8851a71428014a1" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:01.932668 32532 raft_consensus.cc:2454] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:01.932693 32400 raft_consensus.cc:2454] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:01.933059 32205 leader_election.cc:302] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:01.933312 743 raft_consensus.cc:2781] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.933416 743 raft_consensus.cc:482] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.933497 743 raft_consensus.cc:3037] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.935343 743 raft_consensus.cc:504] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.935873 743 leader_election.cc:288] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:01.936348 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130b43c1e784400fb8851a71428014a1" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:01.936422 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130b43c1e784400fb8851a71428014a1" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:01.936597 32532 raft_consensus.cc:3037] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.936596 32400 raft_consensus.cc:3037] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.938403 32532 raft_consensus.cc:2454] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:01.938529 32400 raft_consensus.cc:2454] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:01.938759 32206 leader_election.cc:302] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:13:01.939074 743 raft_consensus.cc:2781] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.939203 743 raft_consensus.cc:686] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:01.939435 743 consensus_queue.cc:237] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:01.941854 31995 catalog_manager.cc:5484] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:01.943756 820 raft_consensus.cc:482] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:01.943895 820 raft_consensus.cc:504] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:01.944701 820 leader_election.cc:288] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:01.947665 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd736780ea840ab9b6630a4acd80ae9" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:01.947888 32532 raft_consensus.cc:2454] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:01.948433 32075 leader_election.cc:302] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:01.948685 820 raft_consensus.cc:2781] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:01.948781 820 raft_consensus.cc:482] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:01.948871 820 raft_consensus.cc:3037] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.951658 820 raft_consensus.cc:504] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:01.952226 820 leader_election.cc:288] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:01.952999 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd736780ea840ab9b6630a4acd80ae9" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:01.953190 32532 raft_consensus.cc:3037] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.953356 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd736780ea840ab9b6630a4acd80ae9" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:01.953382 32272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd736780ea840ab9b6630a4acd80ae9" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:01.953768 32271 raft_consensus.cc:3037] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:01.955906 32532 raft_consensus.cc:2454] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:01.956003 32271 raft_consensus.cc:2454] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:01.956518 32075 leader_election.cc:302] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:01.956931 820 raft_consensus.cc:2781] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:01.957058 820 raft_consensus.cc:686] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Leader. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:01.957319 820 consensus_queue.cc:237] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:01.960263 31995 catalog_manager.cc:5484] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:02.140977 752 raft_consensus.cc:482] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:02.141180 752 raft_consensus.cc:504] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:02.142729 752 leader_election.cc:288] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:02.149366 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734d2de8d2804b2c8d2f144157e12f77" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:02.149379 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734d2de8d2804b2c8d2f144157e12f77" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:02.149587 32271 raft_consensus.cc:2454] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:02.149600 32141 raft_consensus.cc:2454] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:02.150038 32465 leader_election.cc:302] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:02.150416 752 raft_consensus.cc:2781] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:02.150535 752 raft_consensus.cc:482] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:02.150630 752 raft_consensus.cc:3037] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.152499 752 raft_consensus.cc:504] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:02.153064 752 leader_election.cc:288] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:02.153491 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734d2de8d2804b2c8d2f144157e12f77" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:02.153470 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734d2de8d2804b2c8d2f144157e12f77" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:02.153683 32271 raft_consensus.cc:3037] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.153683 32141 raft_consensus.cc:3037] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:02.156313 32141 raft_consensus.cc:2454] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:02.156313 32271 raft_consensus.cc:2454] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:02.156694 32466 leader_election.cc:302] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:13:02.156949 752 raft_consensus.cc:2781] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:02.157287 752 raft_consensus.cc:686] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Becoming Leader. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Running, Role: LEADER
I20240503 08:13:02.157716 752 consensus_queue.cc:237] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:02.161031 31995 catalog_manager.cc:5484] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:02.316987 743 consensus_queue.cc:1035] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.321847 743 consensus_queue.cc:1035] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.459733 820 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:02.466341 860 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:02.539714 853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:02.540030 853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:02.547554 853 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:02.572991 752 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 08:13:02.578194 752 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 210.762 ms
time per row: 0.105381 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:13:03.756150 31996 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:03.764992 32251 tablet_service.cc:1467] Processing CreateTablet for tablet 400a0317f9184c99a449f7f160fa998f (DEFAULT_TABLE table=rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.765074 32381 tablet_service.cc:1467] Processing CreateTablet for tablet 400a0317f9184c99a449f7f160fa998f (DEFAULT_TABLE table=rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.765481 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400a0317f9184c99a449f7f160fa998f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.765481 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400a0317f9184c99a449f7f160fa998f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.765563 32510 tablet_service.cc:1467] Processing CreateTablet for tablet feee2677e29f4614825042f6fb663389 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.765964 32510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feee2677e29f4614825042f6fb663389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.767253 32121 tablet_service.cc:1467] Processing CreateTablet for tablet b7ed99e3dd124de09353f6ed509f3eb1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.767611 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ed99e3dd124de09353f6ed509f3eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.770604 32382 tablet_service.cc:1467] Processing CreateTablet for tablet feee2677e29f4614825042f6fb663389 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.771221 32380 tablet_service.cc:1467] Processing CreateTablet for tablet b7ed99e3dd124de09353f6ed509f3eb1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.771559 32380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ed99e3dd124de09353f6ed509f3eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.772747 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feee2677e29f4614825042f6fb663389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.773212 32120 tablet_service.cc:1467] Processing CreateTablet for tablet 400a0317f9184c99a449f7f160fa998f (DEFAULT_TABLE table=rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.773329 32511 tablet_service.cc:1467] Processing CreateTablet for tablet b7ed99e3dd124de09353f6ed509f3eb1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.773589 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400a0317f9184c99a449f7f160fa998f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.773658 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ed99e3dd124de09353f6ed509f3eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.777411 32252 tablet_service.cc:1467] Processing CreateTablet for tablet feee2677e29f4614825042f6fb663389 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:03.777815 32252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feee2677e29f4614825042f6fb663389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.782228 914 tablet_bootstrap.cc:492] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:03.783689 915 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:03.783812 914 tablet_bootstrap.cc:654] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.785455 915 tablet_bootstrap.cc:654] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.789510 916 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:03.791172 916 tablet_bootstrap.cc:654] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.793452 915 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:03.793608 915 ts_tablet_manager.cc:1367] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240503 08:13:03.794402 915 raft_consensus.cc:348] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:03.794643 915 raft_consensus.cc:374] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.794732 915 raft_consensus.cc:729] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.795002 915 consensus_queue.cc:260] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:03.795720 921 tablet_bootstrap.cc:492] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:03.796182 916 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:03.796327 916 ts_tablet_manager.cc:1367] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:13:03.797186 916 raft_consensus.cc:348] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:03.797406 916 raft_consensus.cc:374] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.797499 916 raft_consensus.cc:729] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.797708 916 consensus_queue.cc:260] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:03.798096 921 tablet_bootstrap.cc:654] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.798223 916 ts_tablet_manager.cc:1398] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:03.798496 916 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:03.798815 915 ts_tablet_manager.cc:1398] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:13:03.800086 916 tablet_bootstrap.cc:654] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.801285 915 tablet_bootstrap.cc:492] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:03.802858 915 tablet_bootstrap.cc:654] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.804630 916 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:03.804786 916 ts_tablet_manager.cc:1367] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 08:13:03.804936 914 tablet_bootstrap.cc:492] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:03.805097 914 ts_tablet_manager.cc:1367] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.009s
I20240503 08:13:03.805473 916 raft_consensus.cc:348] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:03.805686 916 raft_consensus.cc:374] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.805768 916 raft_consensus.cc:729] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.805836 914 raft_consensus.cc:348] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:03.806016 914 raft_consensus.cc:374] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.806106 914 raft_consensus.cc:729] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.806017 916 consensus_queue.cc:260] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:03.806581 916 ts_tablet_manager.cc:1398] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:03.806342 914 consensus_queue.cc:260] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:03.809121 915 tablet_bootstrap.cc:492] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:03.809286 915 ts_tablet_manager.cc:1367] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:13:03.807106 914 ts_tablet_manager.cc:1398] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:03.809996 915 raft_consensus.cc:348] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:03.810206 915 raft_consensus.cc:374] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.810235 914 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:03.810267 915 raft_consensus.cc:729] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.810550 915 consensus_queue.cc:260] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:03.811057 915 ts_tablet_manager.cc:1398] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:03.811333 915 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:03.812980 915 tablet_bootstrap.cc:654] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.814512 914 tablet_bootstrap.cc:654] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.814821 921 tablet_bootstrap.cc:492] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:03.815009 921 ts_tablet_manager.cc:1367] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.006s
I20240503 08:13:03.815779 921 raft_consensus.cc:348] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:03.815994 921 raft_consensus.cc:374] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.816071 921 raft_consensus.cc:729] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.816318 921 consensus_queue.cc:260] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:03.816951 921 ts_tablet_manager.cc:1398] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:03.817240 921 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:03.818898 921 tablet_bootstrap.cc:654] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:03.820850 921 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:03.820988 921 ts_tablet_manager.cc:1367] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:13:03.821491 825 raft_consensus.cc:482] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:03.821627 825 raft_consensus.cc:504] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:03.821756 915 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:03.821871 915 ts_tablet_manager.cc:1367] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240503 08:13:03.822250 825 leader_election.cc:288] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:03.822595 915 raft_consensus.cc:348] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:03.822782 915 raft_consensus.cc:374] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.822898 915 raft_consensus.cc:729] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.822978 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee2677e29f4614825042f6fb663389" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:03.823184 32532 raft_consensus.cc:2454] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:03.823341 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee2677e29f4614825042f6fb663389" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:03.823314 915 consensus_queue.cc:260] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:03.823557 32400 raft_consensus.cc:2454] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:03.823585 921 raft_consensus.cc:348] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:03.823858 921 raft_consensus.cc:374] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.823984 921 raft_consensus.cc:729] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.824359 915 ts_tablet_manager.cc:1398] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:03.823968 32205 leader_election.cc:302] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:03.824328 921 consensus_queue.cc:260] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:03.824913 825 raft_consensus.cc:2781] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:03.825040 825 raft_consensus.cc:482] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:03.825145 825 raft_consensus.cc:3037] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.825781 921 ts_tablet_manager.cc:1398] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:13:03.826135 914 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:03.826289 914 ts_tablet_manager.cc:1367] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240503 08:13:03.827023 914 raft_consensus.cc:348] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:03.827237 914 raft_consensus.cc:374] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:03.827316 914 raft_consensus.cc:729] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:03.827311 825 raft_consensus.cc:504] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:03.827543 914 consensus_queue.cc:260] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:03.827997 914 ts_tablet_manager.cc:1398] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:03.828459 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee2677e29f4614825042f6fb663389" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:03.828624 32532 raft_consensus.cc:3037] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.828842 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee2677e29f4614825042f6fb663389" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:03.829196 32400 raft_consensus.cc:3037] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.831138 32532 raft_consensus.cc:2454] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:03.831980 32400 raft_consensus.cc:2454] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:03.832393 32205 leader_election.cc:302] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:03.832633 928 raft_consensus.cc:2781] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:03.832824 825 leader_election.cc:288] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:03.834904 928 raft_consensus.cc:686] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:03.835147 928 consensus_queue.cc:237] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:03.837450 31996 catalog_manager.cc:5484] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:03.840106 909 raft_consensus.cc:482] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:03.840263 909 raft_consensus.cc:504] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:03.840833 909 leader_election.cc:288] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:03.841171 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ed99e3dd124de09353f6ed509f3eb1" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:03.841225 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ed99e3dd124de09353f6ed509f3eb1" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:03.841375 32400 raft_consensus.cc:2454] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:03.841435 32532 raft_consensus.cc:2454] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:03.841707 32074 leader_election.cc:302] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:03.841924 909 raft_consensus.cc:2781] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:03.842041 909 raft_consensus.cc:482] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:03.842149 909 raft_consensus.cc:3037] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.843894 909 raft_consensus.cc:504] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:03.844395 909 leader_election.cc:288] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Requested vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:03.844664 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ed99e3dd124de09353f6ed509f3eb1" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:03.844784 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ed99e3dd124de09353f6ed509f3eb1" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:03.844869 32400 raft_consensus.cc:3037] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.844972 32532 raft_consensus.cc:3037] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.846583 32400 raft_consensus.cc:2454] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:03.846712 32532 raft_consensus.cc:2454] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:03.846944 32074 leader_election.cc:302] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:03.847222 909 raft_consensus.cc:2781] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:03.847362 909 raft_consensus.cc:686] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Leader. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:03.847621 909 consensus_queue.cc:237] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:03.849821 31996 catalog_manager.cc:5484] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:03.901293 928 raft_consensus.cc:482] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:03.901494 928 raft_consensus.cc:504] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:03.902027 928 leader_election.cc:288] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:03.902549 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a0317f9184c99a449f7f160fa998f" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:03.902662 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a0317f9184c99a449f7f160fa998f" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:03.902768 32400 raft_consensus.cc:2454] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:03.902907 32141 raft_consensus.cc:2454] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:03.903230 32205 leader_election.cc:302] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:03.903520 928 raft_consensus.cc:2781] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:03.903618 928 raft_consensus.cc:482] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:03.903721 928 raft_consensus.cc:3037] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.905494 928 raft_consensus.cc:504] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:03.906016 928 leader_election.cc:288] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:03.906417 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a0317f9184c99a449f7f160fa998f" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:03.906453 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400a0317f9184c99a449f7f160fa998f" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:03.906581 32400 raft_consensus.cc:3037] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.906601 32141 raft_consensus.cc:3037] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:03.908378 32400 raft_consensus.cc:2454] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:03.908414 32141 raft_consensus.cc:2454] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:03.908744 32205 leader_election.cc:302] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:03.908982 928 raft_consensus.cc:2781] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:03.909138 928 raft_consensus.cc:686] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:03.909369 928 consensus_queue.cc:237] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:03.911944 31996 catalog_manager.cc:5484] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:04.227772 928 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.233443 929 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:04.256610 933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:04.256902 933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:04.264079 933 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 08:13:04.310825 909 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.322837 909 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.362692 937 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:04.394073 929 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 231.332 ms
time per row: 0.115666 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:13:05.497663 31991 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:05.505964 32252 tablet_service.cc:1467] Processing CreateTablet for tablet b44429c4434d47c090f9857389c70702 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.506412 32252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44429c4434d47c090f9857389c70702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.507467 32120 tablet_service.cc:1467] Processing CreateTablet for tablet b44429c4434d47c090f9857389c70702 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.507563 32382 tablet_service.cc:1467] Processing CreateTablet for tablet e4e25ba167a248ce883fbafc4f6f5fd6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.507817 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44429c4434d47c090f9857389c70702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.507828 32380 tablet_service.cc:1467] Processing CreateTablet for tablet af5c3865ca694bf090dce19f1100e5eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.507972 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e25ba167a248ce883fbafc4f6f5fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.508232 32380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5c3865ca694bf090dce19f1100e5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.511468 32510 tablet_service.cc:1467] Processing CreateTablet for tablet b44429c4434d47c090f9857389c70702 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.511550 32121 tablet_service.cc:1467] Processing CreateTablet for tablet af5c3865ca694bf090dce19f1100e5eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.511883 32510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44429c4434d47c090f9857389c70702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.511893 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5c3865ca694bf090dce19f1100e5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.515249 32511 tablet_service.cc:1467] Processing CreateTablet for tablet e4e25ba167a248ce883fbafc4f6f5fd6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.515595 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e25ba167a248ce883fbafc4f6f5fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.519884 32250 tablet_service.cc:1467] Processing CreateTablet for tablet e4e25ba167a248ce883fbafc4f6f5fd6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.519986 32251 tablet_service.cc:1467] Processing CreateTablet for tablet af5c3865ca694bf090dce19f1100e5eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:05.520320 32250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e25ba167a248ce883fbafc4f6f5fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.520531 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5c3865ca694bf090dce19f1100e5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.524912 1000 tablet_bootstrap.cc:492] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:05.526677 1001 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:05.526886 1000 tablet_bootstrap.cc:654] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.529413 1001 tablet_bootstrap.cc:654] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.533910 999 tablet_bootstrap.cc:492] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:05.536000 999 tablet_bootstrap.cc:654] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.542574 1001 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:05.542737 1001 ts_tablet_manager.cc:1367] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.013s
I20240503 08:13:05.543442 1001 raft_consensus.cc:348] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:05.543736 1001 raft_consensus.cc:374] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.543859 1001 raft_consensus.cc:729] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.543998 999 tablet_bootstrap.cc:492] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:05.544111 1001 consensus_queue.cc:260] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:05.544242 999 ts_tablet_manager.cc:1367] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240503 08:13:05.544905 999 raft_consensus.cc:348] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:05.545212 999 raft_consensus.cc:374] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.545310 999 raft_consensus.cc:729] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.545339 998 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:05.545498 999 consensus_queue.cc:260] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:05.546068 999 ts_tablet_manager.cc:1398] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:05.546243 1000 tablet_bootstrap.cc:492] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:05.546330 999 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:05.546378 1000 ts_tablet_manager.cc:1367] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20240503 08:13:05.547204 1001 ts_tablet_manager.cc:1398] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:13:05.547129 1000 raft_consensus.cc:348] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:05.547322 1000 raft_consensus.cc:374] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.547412 1000 raft_consensus.cc:729] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.547633 1001 tablet_bootstrap.cc:492] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:05.547739 999 tablet_bootstrap.cc:654] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.549609 1001 tablet_bootstrap.cc:654] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.553337 999 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:05.553470 999 ts_tablet_manager.cc:1367] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 08:13:05.553967 999 raft_consensus.cc:348] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:05.554128 999 raft_consensus.cc:374] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.554198 999 raft_consensus.cc:729] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.554411 999 consensus_queue.cc:260] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:05.554960 999 ts_tablet_manager.cc:1398] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:13:05.555250 999 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:05.556210 998 tablet_bootstrap.cc:654] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.556438 1000 consensus_queue.cc:260] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:05.556649 999 tablet_bootstrap.cc:654] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.557327 1001 tablet_bootstrap.cc:492] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:05.557523 1001 ts_tablet_manager.cc:1367] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240503 08:13:05.557719 1000 ts_tablet_manager.cc:1398] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:13:05.558403 1000 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:05.558331 1001 raft_consensus.cc:348] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:05.558640 1001 raft_consensus.cc:374] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.558794 1001 raft_consensus.cc:729] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.559127 1001 consensus_queue.cc:260] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:05.559934 1000 tablet_bootstrap.cc:654] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.560276 1001 ts_tablet_manager.cc:1398] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:13:05.561647 998 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:05.561785 998 ts_tablet_manager.cc:1367] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 08:13:05.562613 998 raft_consensus.cc:348] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:05.562805 998 raft_consensus.cc:374] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.562942 998 raft_consensus.cc:729] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.563208 998 consensus_queue.cc:260] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:05.563582 1000 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:05.563776 1000 ts_tablet_manager.cc:1367] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:13:05.564313 998 ts_tablet_manager.cc:1398] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:05.564496 999 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:05.564599 998 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:05.564477 1000 raft_consensus.cc:348] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:05.564653 999 ts_tablet_manager.cc:1367] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240503 08:13:05.564808 1000 raft_consensus.cc:374] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.564935 1000 raft_consensus.cc:729] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.565196 1000 consensus_queue.cc:260] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:05.565724 1000 ts_tablet_manager.cc:1398] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:05.566067 998 tablet_bootstrap.cc:654] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:05.569132 999 raft_consensus.cc:348] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:05.569366 999 raft_consensus.cc:374] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.569489 999 raft_consensus.cc:729] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.569732 999 consensus_queue.cc:260] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:05.570399 999 ts_tablet_manager.cc:1398] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:13:05.575423 998 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:05.575585 998 ts_tablet_manager.cc:1367] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240503 08:13:05.576153 998 raft_consensus.cc:348] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:05.576339 998 raft_consensus.cc:374] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:05.576408 998 raft_consensus.cc:729] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:05.576673 998 consensus_queue.cc:260] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:05.576903 1010 raft_consensus.cc:482] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:05.577055 1010 raft_consensus.cc:504] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:05.577459 998 ts_tablet_manager.cc:1398] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:05.578578 1010 leader_election.cc:288] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:05.585670 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5c3865ca694bf090dce19f1100e5eb" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:05.585944 32141 raft_consensus.cc:2454] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 0.
I20240503 08:13:05.586493 32335 leader_election.cc:302] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:05.586905 1010 raft_consensus.cc:2781] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:05.587011 1010 raft_consensus.cc:482] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:05.587100 1010 raft_consensus.cc:3037] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.588963 1010 raft_consensus.cc:504] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:05.589499 1010 leader_election.cc:288] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:05.589936 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5c3865ca694bf090dce19f1100e5eb" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:05.590163 32141 raft_consensus.cc:3037] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.592614 32141 raft_consensus.cc:2454] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 1.
I20240503 08:13:05.592963 32335 leader_election.cc:302] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:05.593235 1010 raft_consensus.cc:2781] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:05.593385 1010 raft_consensus.cc:686] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Becoming Leader. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Running, Role: LEADER
I20240503 08:13:05.593961 1010 consensus_queue.cc:237] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:05.596628 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5c3865ca694bf090dce19f1100e5eb" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:05.596623 32272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5c3865ca694bf090dce19f1100e5eb" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:05.596882 32271 raft_consensus.cc:3037] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.597738 31991 catalog_manager.cc:5484] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:05.598781 32271 raft_consensus.cc:2454] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 1.
I20240503 08:13:05.642716 860 raft_consensus.cc:482] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:05.642931 860 raft_consensus.cc:504] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:05.643491 860 leader_election.cc:288] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:05.643950 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44429c4434d47c090f9857389c70702" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:05.644089 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44429c4434d47c090f9857389c70702" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:05.644176 32271 raft_consensus.cc:2454] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:05.644316 32532 raft_consensus.cc:2454] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:05.644555 32075 leader_election.cc:302] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:05.644811 860 raft_consensus.cc:2781] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:05.644927 860 raft_consensus.cc:482] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:05.645032 860 raft_consensus.cc:3037] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.646806 860 raft_consensus.cc:504] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:05.647333 860 leader_election.cc:288] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:05.647723 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44429c4434d47c090f9857389c70702" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:05.647827 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44429c4434d47c090f9857389c70702" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:05.647913 32271 raft_consensus.cc:3037] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.648015 32532 raft_consensus.cc:3037] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.649818 32532 raft_consensus.cc:2454] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:05.649900 32271 raft_consensus.cc:2454] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:05.650209 32075 leader_election.cc:302] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:05.650480 860 raft_consensus.cc:2781] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:05.650803 860 raft_consensus.cc:686] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Leader. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:05.651083 860 consensus_queue.cc:237] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:05.653545 31991 catalog_manager.cc:5484] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:05.760239 1010 raft_consensus.cc:482] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:05.760454 1010 raft_consensus.cc:504] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:05.761085 1010 leader_election.cc:288] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:05.761891 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e25ba167a248ce883fbafc4f6f5fd6" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:05.762087 32271 raft_consensus.cc:2454] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 0.
I20240503 08:13:05.762478 32336 leader_election.cc:302] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:05.762763 1010 raft_consensus.cc:2781] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:05.762864 1010 raft_consensus.cc:482] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:05.762998 1010 raft_consensus.cc:3037] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.765290 1010 raft_consensus.cc:504] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:05.765866 1010 leader_election.cc:288] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Requested vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:05.766283 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e25ba167a248ce883fbafc4f6f5fd6" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:05.766494 32271 raft_consensus.cc:3037] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:05.767377 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e25ba167a248ce883fbafc4f6f5fd6" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:05.767477 32531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e25ba167a248ce883fbafc4f6f5fd6" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:05.767616 32532 raft_consensus.cc:2454] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 0.
I20240503 08:13:05.768746 32271 raft_consensus.cc:2454] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 1.
I20240503 08:13:05.769095 32336 leader_election.cc:302] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters: 09a050703a1a4c0a8dfc04e4c2853797
I20240503 08:13:05.769327 1010 raft_consensus.cc:2781] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:05.769518 1010 raft_consensus.cc:686] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Becoming Leader. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Running, Role: LEADER
I20240503 08:13:05.769948 1010 consensus_queue.cc:237] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:05.772450 31991 catalog_manager.cc:5484] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:06.093833 1010 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.099339 1012 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.123152 860 consensus_queue.cc:1035] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.127640 860 consensus_queue.cc:1035] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:06.146270 1022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:06.146562 1022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:06.153875 1022 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:06.157857 32531 raft_consensus.cc:3037] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:06.161343 1012 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:06.165452 1012 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 192.464 ms
time per row: 0.0962318 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:13:07.360572 31991 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:07.368821 32511 tablet_service.cc:1467] Processing CreateTablet for tablet f4bd617fecd1483e944365252edf852c (DEFAULT_TABLE table=rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.369266 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bd617fecd1483e944365252edf852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.370012 32120 tablet_service.cc:1467] Processing CreateTablet for tablet f6b5cbb6708049e6ae32da12bb27368f (DEFAULT_TABLE table=rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.370158 32382 tablet_service.cc:1467] Processing CreateTablet for tablet f4bd617fecd1483e944365252edf852c (DEFAULT_TABLE table=rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.370222 32510 tablet_service.cc:1467] Processing CreateTablet for tablet f6b5cbb6708049e6ae32da12bb27368f (DEFAULT_TABLE table=rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.370429 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b5cbb6708049e6ae32da12bb27368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.370507 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bd617fecd1483e944365252edf852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.370617 32510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b5cbb6708049e6ae32da12bb27368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.371188 32512 tablet_service.cc:1467] Processing CreateTablet for tablet 5de086cc1f0b497f87b055e745a38e0d (DEFAULT_TABLE table=rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.371501 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de086cc1f0b497f87b055e745a38e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.374752 32380 tablet_service.cc:1467] Processing CreateTablet for tablet f6b5cbb6708049e6ae32da12bb27368f (DEFAULT_TABLE table=rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.374781 32119 tablet_service.cc:1467] Processing CreateTablet for tablet 5de086cc1f0b497f87b055e745a38e0d (DEFAULT_TABLE table=rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.375144 32380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b5cbb6708049e6ae32da12bb27368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.375159 32119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de086cc1f0b497f87b055e745a38e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.379216 32121 tablet_service.cc:1467] Processing CreateTablet for tablet f4bd617fecd1483e944365252edf852c (DEFAULT_TABLE table=rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.379551 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bd617fecd1483e944365252edf852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.379768 32252 tablet_service.cc:1467] Processing CreateTablet for tablet 5de086cc1f0b497f87b055e745a38e0d (DEFAULT_TABLE table=rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:07.380183 32252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de086cc1f0b497f87b055e745a38e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:07.385669 1094 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:07.387542 1094 tablet_bootstrap.cc:654] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.392060 1096 tablet_bootstrap.cc:492] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:07.392601 1095 tablet_bootstrap.cc:492] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:07.393774 1094 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:07.393949 1094 ts_tablet_manager.cc:1367] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240503 08:13:07.394479 1095 tablet_bootstrap.cc:654] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.394730 1094 raft_consensus.cc:348] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.395009 1094 raft_consensus.cc:374] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.395143 1094 raft_consensus.cc:729] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.395395 1094 consensus_queue.cc:260] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.395996 1096 tablet_bootstrap.cc:654] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.396142 1094 ts_tablet_manager.cc:1398] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:07.396442 1094 tablet_bootstrap.cc:492] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:07.398239 1094 tablet_bootstrap.cc:654] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.401638 1098 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:07.402386 1095 tablet_bootstrap.cc:492] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:07.402581 1095 ts_tablet_manager.cc:1367] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 08:13:07.403424 1095 raft_consensus.cc:348] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.403688 1095 raft_consensus.cc:374] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.403839 1095 raft_consensus.cc:729] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.404138 1095 consensus_queue.cc:260] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.404803 1095 ts_tablet_manager.cc:1398] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:07.404909 1098 tablet_bootstrap.cc:654] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.405092 1095 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:07.406510 1094 tablet_bootstrap.cc:492] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:07.406544 1095 tablet_bootstrap.cc:654] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.406737 1094 ts_tablet_manager.cc:1367] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 08:13:07.407544 1094 raft_consensus.cc:348] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.407824 1094 raft_consensus.cc:374] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.407931 1094 raft_consensus.cc:729] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.408211 1094 consensus_queue.cc:260] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.409574 1094 ts_tablet_manager.cc:1398] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:07.410693 1098 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:07.410821 1098 ts_tablet_manager.cc:1367] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240503 08:13:07.411216 1096 tablet_bootstrap.cc:492] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:07.411373 1096 ts_tablet_manager.cc:1367] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.007s
I20240503 08:13:07.411566 1098 raft_consensus.cc:348] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:07.411800 1095 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:07.411803 1098 raft_consensus.cc:374] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.411918 1098 raft_consensus.cc:729] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.412040 1095 ts_tablet_manager.cc:1367] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:13:07.412125 1096 raft_consensus.cc:348] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.412158 1098 consensus_queue.cc:260] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:07.412344 1096 raft_consensus.cc:374] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.412456 1096 raft_consensus.cc:729] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.412675 1096 consensus_queue.cc:260] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.413234 1096 ts_tablet_manager.cc:1398] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:07.413566 1096 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:07.413890 1098 ts_tablet_manager.cc:1398] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:07.414100 1095 raft_consensus.cc:348] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:07.414433 1095 raft_consensus.cc:374] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.414547 1095 raft_consensus.cc:729] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.414815 1095 consensus_queue.cc:260] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:07.415189 1096 tablet_bootstrap.cc:654] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.415390 1095 ts_tablet_manager.cc:1398] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:07.415638 1095 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:07.417055 1096 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:07.417203 1096 ts_tablet_manager.cc:1367] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:07.417838 1096 raft_consensus.cc:348] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:07.418047 1096 raft_consensus.cc:374] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.418125 1096 raft_consensus.cc:729] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.418236 1095 tablet_bootstrap.cc:654] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.418445 1096 consensus_queue.cc:260] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:07.419061 1096 ts_tablet_manager.cc:1398] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:07.419327 1096 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:07.420706 1095 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:07.420737 1096 tablet_bootstrap.cc:654] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:07.420840 1095 ts_tablet_manager.cc:1367] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:13:07.421415 1095 raft_consensus.cc:348] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.421586 1095 raft_consensus.cc:374] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.421669 1095 raft_consensus.cc:729] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.421928 1095 consensus_queue.cc:260] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.422545 1095 ts_tablet_manager.cc:1398] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:07.424072 1096 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:07.424176 1096 ts_tablet_manager.cc:1367] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 08:13:07.424772 1096 raft_consensus.cc:348] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.424947 1096 raft_consensus.cc:374] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:07.425035 1096 raft_consensus.cc:729] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:07.425228 1096 consensus_queue.cc:260] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.425669 1096 ts_tablet_manager.cc:1398] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:07.457142 929 raft_consensus.cc:482] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:07.457291 929 raft_consensus.cc:504] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:07.457866 929 leader_election.cc:288] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:07.458308 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de086cc1f0b497f87b055e745a38e0d" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:07.458312 32531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de086cc1f0b497f87b055e745a38e0d" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:07.458503 32531 raft_consensus.cc:2454] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:07.458494 32141 raft_consensus.cc:2454] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:07.458892 32206 leader_election.cc:302] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:13:07.459167 929 raft_consensus.cc:2781] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:07.459275 929 raft_consensus.cc:482] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:07.459353 929 raft_consensus.cc:3037] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.461145 929 raft_consensus.cc:504] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:07.461619 929 leader_election.cc:288] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:07.462011 32531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de086cc1f0b497f87b055e745a38e0d" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:07.462014 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de086cc1f0b497f87b055e745a38e0d" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:07.462159 32531 raft_consensus.cc:3037] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.462179 32141 raft_consensus.cc:3037] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.463937 32141 raft_consensus.cc:2454] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:07.463938 32531 raft_consensus.cc:2454] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:07.464265 32205 leader_election.cc:302] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:07.464533 929 raft_consensus.cc:2781] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:07.464854 929 raft_consensus.cc:686] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:07.465127 929 consensus_queue.cc:237] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:07.467563 31994 catalog_manager.cc:5484] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.536919 1069 raft_consensus.cc:482] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:07.537091 1069 raft_consensus.cc:504] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.537736 1069 leader_election.cc:288] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:07.538188 32531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b5cbb6708049e6ae32da12bb27368f" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:07.538435 32531 raft_consensus.cc:2454] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 0.
I20240503 08:13:07.538457 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b5cbb6708049e6ae32da12bb27368f" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:07.538751 32141 raft_consensus.cc:2454] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 0.
I20240503 08:13:07.538820 32336 leader_election.cc:302] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:07.539088 1069 raft_consensus.cc:2781] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:07.539213 1069 raft_consensus.cc:482] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:07.539304 1069 raft_consensus.cc:3037] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.541409 1069 raft_consensus.cc:504] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.541908 1069 leader_election.cc:288] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:07.542201 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b5cbb6708049e6ae32da12bb27368f" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:07.542390 32141 raft_consensus.cc:3037] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.542356 32531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b5cbb6708049e6ae32da12bb27368f" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:07.542544 32531 raft_consensus.cc:3037] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.544202 32141 raft_consensus.cc:2454] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 1.
I20240503 08:13:07.544528 32335 leader_election.cc:302] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:07.544764 1069 raft_consensus.cc:2781] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:07.545159 1069 raft_consensus.cc:686] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Becoming Leader. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Running, Role: LEADER
I20240503 08:13:07.545225 32531 raft_consensus.cc:2454] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 1.
I20240503 08:13:07.545429 1069 consensus_queue.cc:237] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.547964 31994 catalog_manager.cc:5484] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:07.616796 1069 raft_consensus.cc:482] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:07.617002 1069 raft_consensus.cc:504] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.617660 1069 leader_election.cc:288] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:07.618083 32531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bd617fecd1483e944365252edf852c" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:07.618206 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bd617fecd1483e944365252edf852c" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:07.618302 32531 raft_consensus.cc:2454] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 0.
I20240503 08:13:07.618422 32141 raft_consensus.cc:2454] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 0.
I20240503 08:13:07.618741 32336 leader_election.cc:302] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:07.619067 1069 raft_consensus.cc:2781] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:07.619175 1069 raft_consensus.cc:482] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:07.619261 1069 raft_consensus.cc:3037] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.621158 1069 raft_consensus.cc:504] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.621729 1069 leader_election.cc:288] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Requested vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:07.622054 32531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bd617fecd1483e944365252edf852c" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:07.622265 32531 raft_consensus.cc:3037] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.622244 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bd617fecd1483e944365252edf852c" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:07.622422 32141 raft_consensus.cc:3037] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:07.624370 32141 raft_consensus.cc:2454] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 1.
I20240503 08:13:07.624488 32531 raft_consensus.cc:2454] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 1.
I20240503 08:13:07.624707 32335 leader_election.cc:302] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:07.625032 1069 raft_consensus.cc:2781] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:07.625196 1069 raft_consensus.cc:686] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Becoming Leader. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Running, Role: LEADER
I20240503 08:13:07.625491 1069 consensus_queue.cc:237] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:07.628080 31994 catalog_manager.cc:5484] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:07.855783 929 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:07.861783 929 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:08.000794 1112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:08.001092 1112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:08.008632 1112 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 08:13:08.038810 1069 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.044756 1069 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.137995 1135 consensus_queue.cc:1035] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:08.149406 32532 raft_consensus.cc:1261] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.155603 1109 consensus_queue.cc:1035] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:08.214062 32205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:41731, user_credentials={real_user=slave}} blocked reactor thread for 53503us
INSERT report
rows total: 2000
time total: 228.274 ms
time per row: 0.114137 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:13:09.247926 31994 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:09.257086 32380 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc8a3241af248578b6ce87b1e33cea9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.257292 32120 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc8a3241af248578b6ce87b1e33cea9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.257578 32380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc8a3241af248578b6ce87b1e33cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.257719 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc8a3241af248578b6ce87b1e33cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.258757 32121 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3ee53044cd43debe6b59a2ef51c230 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.259150 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3ee53044cd43debe6b59a2ef51c230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.262146 32251 tablet_service.cc:1467] Processing CreateTablet for tablet 59a99f630f59472ca5c6778525e79fbe (DEFAULT_TABLE table=rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.262240 32250 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3ee53044cd43debe6b59a2ef51c230 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.262537 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a99f630f59472ca5c6778525e79fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.262583 32252 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc8a3241af248578b6ce87b1e33cea9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.262909 32252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc8a3241af248578b6ce87b1e33cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.265045 32250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3ee53044cd43debe6b59a2ef51c230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.267812 32510 tablet_service.cc:1467] Processing CreateTablet for tablet 59a99f630f59472ca5c6778525e79fbe (DEFAULT_TABLE table=rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.268214 32510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a99f630f59472ca5c6778525e79fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.271715 1181 tablet_bootstrap.cc:492] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:09.271780 32382 tablet_service.cc:1467] Processing CreateTablet for tablet 59a99f630f59472ca5c6778525e79fbe (DEFAULT_TABLE table=rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.272161 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a99f630f59472ca5c6778525e79fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.273344 1181 tablet_bootstrap.cc:654] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.275243 32381 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3ee53044cd43debe6b59a2ef51c230 (DEFAULT_TABLE table=rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:09.275628 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3ee53044cd43debe6b59a2ef51c230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.288820 1181 tablet_bootstrap.cc:492] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:09.289017 1181 ts_tablet_manager.cc:1367] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240503 08:13:09.289906 1185 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:09.289947 1181 raft_consensus.cc:348] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:09.290333 1181 raft_consensus.cc:374] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.290491 1181 raft_consensus.cc:729] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.290827 1181 consensus_queue.cc:260] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:09.291535 1181 ts_tablet_manager.cc:1398] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:09.292384 1181 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:09.293004 1183 tablet_bootstrap.cc:492] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:09.294554 1183 tablet_bootstrap.cc:654] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.294626 1185 tablet_bootstrap.cc:654] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.295531 1181 tablet_bootstrap.cc:654] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.295943 1184 tablet_bootstrap.cc:492] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:09.297462 1184 tablet_bootstrap.cc:654] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.298681 1185 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:09.298731 1181 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:09.298918 1181 ts_tablet_manager.cc:1367] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:13:09.299022 1183 tablet_bootstrap.cc:492] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:09.298959 1185 ts_tablet_manager.cc:1367] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240503 08:13:09.299171 1183 ts_tablet_manager.cc:1367] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:13:09.299665 1181 raft_consensus.cc:348] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:09.299902 1181 raft_consensus.cc:374] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.299978 1181 raft_consensus.cc:729] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.299908 1183 raft_consensus.cc:348] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:09.300191 1183 raft_consensus.cc:374] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.300298 1183 raft_consensus.cc:729] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.300019 1185 raft_consensus.cc:348] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:09.300284 1181 consensus_queue.cc:260] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:09.300570 1185 raft_consensus.cc:374] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.300529 1183 consensus_queue.cc:260] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:09.301004 1181 ts_tablet_manager.cc:1398] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:09.301795 1185 raft_consensus.cc:729] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.301870 1183 ts_tablet_manager.cc:1398] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:09.302219 1183 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:09.302189 1185 consensus_queue.cc:260] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:09.304028 1185 ts_tablet_manager.cc:1398] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:09.306370 1183 tablet_bootstrap.cc:654] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.307097 1184 tablet_bootstrap.cc:492] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:09.307247 1184 ts_tablet_manager.cc:1367] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 08:13:09.309195 1184 raft_consensus.cc:348] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:09.309427 1184 raft_consensus.cc:374] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.309556 1184 raft_consensus.cc:729] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.309826 1184 consensus_queue.cc:260] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:09.310590 1184 ts_tablet_manager.cc:1398] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:09.310892 1184 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:09.312748 1184 tablet_bootstrap.cc:654] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.313246 1183 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:09.313405 1183 ts_tablet_manager.cc:1367] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20240503 08:13:09.314077 1183 raft_consensus.cc:348] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:09.314303 1183 raft_consensus.cc:374] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.314371 1183 raft_consensus.cc:729] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.314497 1184 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:09.314642 1184 ts_tablet_manager.cc:1367] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:13:09.314569 1183 consensus_queue.cc:260] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:09.315079 1183 ts_tablet_manager.cc:1398] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:09.315253 1184 raft_consensus.cc:348] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:09.315371 1183 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:09.315412 1184 raft_consensus.cc:374] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.315493 1184 raft_consensus.cc:729] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.315716 1184 consensus_queue.cc:260] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:09.316094 1184 ts_tablet_manager.cc:1398] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:09.316294 1184 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:09.316831 1183 tablet_bootstrap.cc:654] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.317445 1184 tablet_bootstrap.cc:654] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:09.321192 1184 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:09.321321 1184 ts_tablet_manager.cc:1367] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 08:13:09.321482 1174 raft_consensus.cc:482] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:09.321627 1174 raft_consensus.cc:504] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:09.321992 1184 raft_consensus.cc:348] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:09.322166 1184 raft_consensus.cc:374] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.322362 1174 leader_election.cc:288] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:09.322439 1184 raft_consensus.cc:729] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.322635 1184 consensus_queue.cc:260] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:09.323020 32401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:09.323231 32401 raft_consensus.cc:2454] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:09.323637 1184 ts_tablet_manager.cc:1398] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:09.323840 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:09.324110 32271 raft_consensus.cc:2454] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:09.324146 32074 leader_election.cc:302] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:09.324524 1174 raft_consensus.cc:2781] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:09.324645 1174 raft_consensus.cc:482] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:09.324710 1174 raft_consensus.cc:3037] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.326354 1183 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:09.326499 1183 ts_tablet_manager.cc:1367] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 08:13:09.327069 1174 raft_consensus.cc:504] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:09.327253 1183 raft_consensus.cc:348] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:09.327446 1183 raft_consensus.cc:374] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:09.327544 1183 raft_consensus.cc:729] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:09.327776 1183 consensus_queue.cc:260] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:09.327962 32401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:09.328131 32401 raft_consensus.cc:3037] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.328261 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:09.328328 1183 ts_tablet_manager.cc:1398] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:09.328452 32271 raft_consensus.cc:3037] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.327636 1174 leader_election.cc:288] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Requested vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:09.330736 32401 raft_consensus.cc:2454] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:09.331027 32271 raft_consensus.cc:2454] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:09.331104 1088 raft_consensus.cc:482] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:09.331183 32074 leader_election.cc:302] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:09.331255 1088 raft_consensus.cc:504] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:09.331559 1174 raft_consensus.cc:2781] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:09.331653 1174 raft_consensus.cc:686] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Leader. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:09.332105 1088 leader_election.cc:288] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:09.331828 1174 consensus_queue.cc:237] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:09.332528 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a99f630f59472ca5c6778525e79fbe" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:09.332947 32271 raft_consensus.cc:2454] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:09.333349 32466 leader_election.cc:302] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:13:09.333699 1088 raft_consensus.cc:2781] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:09.333819 1088 raft_consensus.cc:482] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:09.333906 1088 raft_consensus.cc:3037] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.336808 1088 raft_consensus.cc:504] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:09.337554 1088 leader_election.cc:288] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:09.338049 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a99f630f59472ca5c6778525e79fbe" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:09.338287 32271 raft_consensus.cc:3037] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.341795 32271 raft_consensus.cc:2454] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:09.342317 32466 leader_election.cc:302] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:13:09.342693 1088 raft_consensus.cc:2781] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:09.347198 31994 catalog_manager.cc:5484] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.350157 1088 raft_consensus.cc:686] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Becoming Leader. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Running, Role: LEADER
I20240503 08:13:09.350473 1088 consensus_queue.cc:237] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:09.352484 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a99f630f59472ca5c6778525e79fbe" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:09.352722 32400 raft_consensus.cc:3037] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.352833 32401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a99f630f59472ca5c6778525e79fbe" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:09.353636 31994 catalog_manager.cc:5484] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.354715 32400 raft_consensus.cc:2454] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:09.541765 1174 raft_consensus.cc:482] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:09.541947 1174 raft_consensus.cc:504] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:09.542544 1174 leader_election.cc:288] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:09.543071 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3ee53044cd43debe6b59a2ef51c230" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:09.543197 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3ee53044cd43debe6b59a2ef51c230" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:09.543316 32271 raft_consensus.cc:2454] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:09.543409 32400 raft_consensus.cc:2454] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:09.543730 32075 leader_election.cc:302] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:09.544013 1174 raft_consensus.cc:2781] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:09.544135 1174 raft_consensus.cc:482] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:09.544272 1174 raft_consensus.cc:3037] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.546983 1174 raft_consensus.cc:504] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:09.547628 1174 leader_election.cc:288] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Requested vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:09.547964 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3ee53044cd43debe6b59a2ef51c230" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:09.548170 32400 raft_consensus.cc:3037] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.548280 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3ee53044cd43debe6b59a2ef51c230" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:09.548851 32271 raft_consensus.cc:3037] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:09.550714 32271 raft_consensus.cc:2454] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:09.551016 32400 raft_consensus.cc:2454] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:09.551076 32075 leader_election.cc:302] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:09.551335 1174 raft_consensus.cc:2781] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:09.551503 1174 raft_consensus.cc:686] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Leader. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:09.551771 1174 consensus_queue.cc:237] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:09.554821 31994 catalog_manager.cc:5484] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:09.735114 1088 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.741086 1197 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.822568 1174 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.829358 1174 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:09.910053 1200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:09.910404 1200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:09.917855 1200 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:09.930728 1089 consensus_queue.cc:1035] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:09.935599 1089 consensus_queue.cc:1035] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 257.196 ms
time per row: 0.128598 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:13:11.186982 31999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:11.195768 32121 tablet_service.cc:1467] Processing CreateTablet for tablet afdc26bd51bd467f88d7892edb81ca9a (DEFAULT_TABLE table=rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:11.195930 32250 tablet_service.cc:1467] Processing CreateTablet for tablet afdc26bd51bd467f88d7892edb81ca9a (DEFAULT_TABLE table=rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:11.196244 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc26bd51bd467f88d7892edb81ca9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.196359 32250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc26bd51bd467f88d7892edb81ca9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.197329 32510 tablet_service.cc:1467] Processing CreateTablet for tablet afdc26bd51bd467f88d7892edb81ca9a (DEFAULT_TABLE table=rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:11.197479 32512 tablet_service.cc:1467] Processing CreateTablet for tablet c970023069d44323a2b38629da1e2fad (DEFAULT_TABLE table=rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:11.197718 32510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc26bd51bd467f88d7892edb81ca9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.198697 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c970023069d44323a2b38629da1e2fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.199689 32381 tablet_service.cc:1467] Processing CreateTablet for tablet 84bef4a86df446d0b4ff490822068fbb (DEFAULT_TABLE table=rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:11.200083 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bef4a86df446d0b4ff490822068fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.202523 32511 tablet_service.cc:1467] Processing CreateTablet for tablet 84bef4a86df446d0b4ff490822068fbb (DEFAULT_TABLE table=rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:11.202904 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bef4a86df446d0b4ff490822068fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.203048 32120 tablet_service.cc:1467] Processing CreateTablet for tablet c970023069d44323a2b38629da1e2fad (DEFAULT_TABLE table=rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:11.203195 32380 tablet_service.cc:1467] Processing CreateTablet for tablet c970023069d44323a2b38629da1e2fad (DEFAULT_TABLE table=rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:11.203442 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c970023069d44323a2b38629da1e2fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.203543 32380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c970023069d44323a2b38629da1e2fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.208189 32251 tablet_service.cc:1467] Processing CreateTablet for tablet 84bef4a86df446d0b4ff490822068fbb (DEFAULT_TABLE table=rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:11.208616 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bef4a86df446d0b4ff490822068fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:11.221943 1266 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:11.225458 1268 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:11.225054 1266 tablet_bootstrap.cc:654] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:11.227453 1269 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:11.227676 1268 tablet_bootstrap.cc:654] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:11.227841 1265 tablet_bootstrap.cc:492] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:11.229511 1265 tablet_bootstrap.cc:654] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:11.229539 1269 tablet_bootstrap.cc:654] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:11.232297 1266 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:11.234098 1266 ts_tablet_manager.cc:1367] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240503 08:13:11.235085 1266 raft_consensus.cc:348] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:11.235448 1266 raft_consensus.cc:374] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:11.235594 1266 raft_consensus.cc:729] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:11.235694 1268 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:11.235978 1268 ts_tablet_manager.cc:1367] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240503 08:13:11.235968 1266 consensus_queue.cc:260] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:11.236865 1268 raft_consensus.cc:348] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:11.237133 1268 raft_consensus.cc:374] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:11.237278 1268 raft_consensus.cc:729] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:11.237579 1268 consensus_queue.cc:260] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:11.237838 1265 tablet_bootstrap.cc:492] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:11.237977 1265 ts_tablet_manager.cc:1367] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 08:13:11.238703 1265 raft_consensus.cc:348] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:11.238966 1268 ts_tablet_manager.cc:1398] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:13:11.238994 1265 raft_consensus.cc:374] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:11.239090 1265 raft_consensus.cc:729] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:11.239303 1265 consensus_queue.cc:260] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:11.239426 1268 tablet_bootstrap.cc:492] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:11.239789 1265 ts_tablet_manager.cc:1398] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:11.240053 1265 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:11.241559 1265 tablet_bootstrap.cc:654] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:11.241724 1268 tablet_bootstrap.cc:654] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:11.248311 1268 tablet_bootstrap.cc:492] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:11.248549 1268 ts_tablet_manager.cc:1367] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240503 08:13:11.248891 1266 ts_tablet_manager.cc:1398] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 08:13:11.248971 1265 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:11.249186 1265 ts_tablet_manager.cc:1367] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 08:13:11.249481 1268 raft_consensus.cc:348] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:11.249728 1268 raft_consensus.cc:374] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:11.249835 1268 raft_consensus.cc:729] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:11.250022 1265 raft_consensus.cc:348] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:11.250130 1268 consensus_queue.cc:260] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:11.250319 1265 raft_consensus.cc:374] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:11.250425 1265 raft_consensus.cc:729] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:11.251039 1265 consensus_queue.cc:260] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:11.251286 1268 ts_tablet_manager.cc:1398] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:11.251734 1265 ts_tablet_manager.cc:1398] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:11.253087 1266 tablet_bootstrap.cc:492] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:11.253258 1197 raft_consensus.cc:482] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:11.253466 1197 raft_consensus.cc:504] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:11.254155 1197 leader_election.cc:288] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:11.255005 1266 tablet_bootstrap.cc:654] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:11.255134 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c970023069d44323a2b38629da1e2fad" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:11.255383 32141 raft_consensus.cc:2454] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:11.255824 32465 leader_election.cc:302] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:11.256114 1197 raft_consensus.cc:2781] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:11.256238 1197 raft_consensus.cc:482] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:11.256353 1197 raft_consensus.cc:3037] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:11.257192 1269 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:11.257341 1269 ts_tablet_manager.cc:1367] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.023s
I20240503 08:13:11.258158 1269 raft_consensus.cc:348] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:11.258383 1269 raft_consensus.cc:374] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:11.258468 1269 raft_consensus.cc:729] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:11.258672 1269 consensus_queue.cc:260] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:11.259104 32402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c970023069d44323a2b38629da1e2fad" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:11.259274 1269 ts_tablet_manager.cc:1398] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:11.259537 1269 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:11.260085 1197 raft_consensus.cc:504] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
W20240503 08:13:11.260466 32465 leader_election.cc:341] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:11.260792 1197 leader_election.cc:288] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Requested vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:11.261230 1269 tablet_bootstrap.cc:654] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:11.261363 32402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c970023069d44323a2b38629da1e2fad" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:11.261493 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c970023069d44323a2b38629da1e2fad" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:11.261718 32141 raft_consensus.cc:3037] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:13:11.261921 32465 leader_election.cc:341] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:11.263890 1266 tablet_bootstrap.cc:492] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:11.264039 1266 ts_tablet_manager.cc:1367] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240503 08:13:11.264309 32141 raft_consensus.cc:2454] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:11.264654 1269 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:11.264719 32465 leader_election.cc:302] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters: 71ad6bf07682497aaa1e9429320aef79
I20240503 08:13:11.264797 1269 ts_tablet_manager.cc:1367] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:13:11.264818 1266 raft_consensus.cc:348] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:11.265002 1266 raft_consensus.cc:374] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:11.265070 1266 raft_consensus.cc:729] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:11.265112 1197 raft_consensus.cc:2781] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:11.265307 1266 consensus_queue.cc:260] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:11.265375 1269 raft_consensus.cc:348] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:11.265594 1269 raft_consensus.cc:374] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:11.265661 1269 raft_consensus.cc:729] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:11.265867 1269 consensus_queue.cc:260] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:11.266345 1269 ts_tablet_manager.cc:1398] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:11.266906 1197 raft_consensus.cc:686] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Becoming Leader. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Running, Role: LEADER
I20240503 08:13:11.267189 1197 consensus_queue.cc:237] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:11.275329 1266 ts_tablet_manager.cc:1398] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.011s user 0.003s sys 0.006s
I20240503 08:13:11.275848 1266 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:11.277601 1266 tablet_bootstrap.cc:654] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:11.277499 31994 catalog_manager.cc:5484] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:11.279323 1266 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:11.279461 1266 ts_tablet_manager.cc:1367] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:13:11.280084 1266 raft_consensus.cc:348] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:11.280267 1266 raft_consensus.cc:374] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:11.280402 1266 raft_consensus.cc:729] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:11.280615 1266 consensus_queue.cc:260] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:11.281050 1266 ts_tablet_manager.cc:1398] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:13:11.308727 1267 raft_consensus.cc:482] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:11.308904 1267 raft_consensus.cc:504] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:11.309542 1267 leader_election.cc:288] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:11.310179 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc26bd51bd467f88d7892edb81ca9a" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:11.310171 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc26bd51bd467f88d7892edb81ca9a" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:11.310415 32141 raft_consensus.cc:2454] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:11.310415 32532 raft_consensus.cc:2454] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:11.310803 32205 leader_election.cc:302] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:11.311087 1267 raft_consensus.cc:2781] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:11.311230 1267 raft_consensus.cc:482] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:11.311328 1267 raft_consensus.cc:3037] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:11.313197 1267 raft_consensus.cc:504] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:11.313740 1267 leader_election.cc:288] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:11.314177 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc26bd51bd467f88d7892edb81ca9a" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:11.314342 32141 raft_consensus.cc:3037] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:11.314275 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc26bd51bd467f88d7892edb81ca9a" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:11.314464 32532 raft_consensus.cc:3037] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:11.316229 32141 raft_consensus.cc:2454] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:11.316468 32532 raft_consensus.cc:2454] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:11.316593 32205 leader_election.cc:302] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:11.316864 1267 raft_consensus.cc:2781] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:11.316980 1267 raft_consensus.cc:686] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:11.317224 1267 consensus_queue.cc:237] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:11.320243 31994 catalog_manager.cc:5484] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:11.330998 1197 raft_consensus.cc:482] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:11.331236 1197 raft_consensus.cc:504] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:11.331910 1197 leader_election.cc:288] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:11.332391 32272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bef4a86df446d0b4ff490822068fbb" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:11.332377 32402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bef4a86df446d0b4ff490822068fbb" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:11.332607 32272 raft_consensus.cc:2454] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:11.332634 32402 raft_consensus.cc:2454] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:11.332998 32465 leader_election.cc:302] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:11.333253 1197 raft_consensus.cc:2781] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:11.333379 1197 raft_consensus.cc:482] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:11.333490 1197 raft_consensus.cc:3037] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:11.335664 1197 raft_consensus.cc:504] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:11.336154 1197 leader_election.cc:288] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Requested vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:11.336617 32402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bef4a86df446d0b4ff490822068fbb" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:11.336613 32272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bef4a86df446d0b4ff490822068fbb" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:11.336781 32272 raft_consensus.cc:3037] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:11.336812 32402 raft_consensus.cc:3037] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:11.338552 32272 raft_consensus.cc:2454] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:11.338655 32402 raft_consensus.cc:2454] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:11.338963 32466 leader_election.cc:302] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:13:11.339270 1197 raft_consensus.cc:2781] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:11.339393 1197 raft_consensus.cc:686] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Becoming Leader. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Running, Role: LEADER
I20240503 08:13:11.339674 1197 consensus_queue.cc:237] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } }
I20240503 08:13:11.342293 31994 catalog_manager.cc:5484] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:11.434015 1069 raft_consensus.cc:482] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:11.434198 1069 raft_consensus.cc:504] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:11.434918 1069 leader_election.cc:288] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:11.435549 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c970023069d44323a2b38629da1e2fad" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:11.435549 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c970023069d44323a2b38629da1e2fad" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:11.435879 32141 raft_consensus.cc:2379] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71ad6bf07682497aaa1e9429320aef79 in current term 1: Already voted for candidate 09a050703a1a4c0a8dfc04e4c2853797 in this term.
I20240503 08:13:11.436362 32335 leader_election.cc:302] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79; no voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90
I20240503 08:13:11.436671 1069 raft_consensus.cc:3037] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:11.439571 1069 raft_consensus.cc:2726] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:11.693547 1197 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:11.699234 1197 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:11.700192 1285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:11.700590 1285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:13:11.704434 1287 consensus_queue.cc:1035] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:11.713061 1285 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:11.738337 1287 consensus_queue.cc:1035] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
Using table 'rebalance_test_table_7'
I20240503 08:13:11.800485 1197 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:11.806725 1279 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 247.383 ms
time per row: 0.123692 ms
I20240503 08:13:12.154814 31995 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:12.163220 32121 tablet_service.cc:1467] Processing CreateTablet for tablet d9151dcb76174b27991fdcf3839db0a5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:12.163522 32251 tablet_service.cc:1467] Processing CreateTablet for tablet d9151dcb76174b27991fdcf3839db0a5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:12.163676 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9151dcb76174b27991fdcf3839db0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.163805 32512 tablet_service.cc:1467] Processing CreateTablet for tablet d9151dcb76174b27991fdcf3839db0a5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:12.163921 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9151dcb76174b27991fdcf3839db0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.164167 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9151dcb76174b27991fdcf3839db0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.167991 32381 tablet_service.cc:1467] Processing CreateTablet for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:12.168154 32250 tablet_service.cc:1467] Processing CreateTablet for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:12.168380 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.168536 32250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.171281 32120 tablet_service.cc:1467] Processing CreateTablet for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:12.171670 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.175195 32252 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf32f9e1e914653a6594d58c67cd80e (DEFAULT_TABLE table=rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:12.175298 32380 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf32f9e1e914653a6594d58c67cd80e (DEFAULT_TABLE table=rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:12.175611 32252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf32f9e1e914653a6594d58c67cd80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.175674 32380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf32f9e1e914653a6594d58c67cd80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.179528 1345 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:12.179659 32511 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf32f9e1e914653a6594d58c67cd80e (DEFAULT_TABLE table=rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:12.180048 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf32f9e1e914653a6594d58c67cd80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:12.181097 1345 tablet_bootstrap.cc:654] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:12.183982 1347 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:12.185935 1347 tablet_bootstrap.cc:654] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:12.186180 1344 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:12.188162 1344 tablet_bootstrap.cc:654] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:12.190025 1344 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:12.190218 1344 ts_tablet_manager.cc:1367] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:13:12.191095 1344 raft_consensus.cc:348] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:12.191394 1344 raft_consensus.cc:374] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:12.191502 1344 raft_consensus.cc:729] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:12.191726 1344 consensus_queue.cc:260] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:12.192418 1344 ts_tablet_manager.cc:1398] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:12.192793 1344 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:12.194368 1344 tablet_bootstrap.cc:654] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:12.195771 1350 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:12.197969 1350 tablet_bootstrap.cc:654] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:12.198549 1347 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:12.198745 1347 ts_tablet_manager.cc:1367] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 08:13:12.199642 1347 raft_consensus.cc:348] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:12.199909 1347 raft_consensus.cc:374] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:12.200038 1347 raft_consensus.cc:729] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:12.200326 1344 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:12.200505 1344 ts_tablet_manager.cc:1367] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:13:12.200517 1345 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:12.200657 1345 ts_tablet_manager.cc:1367] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.006s
I20240503 08:13:12.201229 1344 raft_consensus.cc:348] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:12.201401 1350 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:12.201476 1344 raft_consensus.cc:374] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:12.201578 1350 ts_tablet_manager.cc:1367] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 08:13:12.201575 1344 raft_consensus.cc:729] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:12.201397 1347 consensus_queue.cc:260] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:12.201792 1344 consensus_queue.cc:260] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:12.201684 1345 raft_consensus.cc:348] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:12.202227 1345 raft_consensus.cc:374] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:12.202287 1344 ts_tablet_manager.cc:1398] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:12.202342 1345 raft_consensus.cc:729] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:12.202335 1350 raft_consensus.cc:348] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:12.202545 1350 raft_consensus.cc:374] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:12.202600 1344 tablet_bootstrap.cc:492] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:12.202651 1350 raft_consensus.cc:729] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:12.202565 1345 consensus_queue.cc:260] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:12.202958 1350 consensus_queue.cc:260] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:12.203576 1350 ts_tablet_manager.cc:1398] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:12.203847 1350 tablet_bootstrap.cc:492] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:12.204181 1345 ts_tablet_manager.cc:1398] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:13:12.204259 1344 tablet_bootstrap.cc:654] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:12.204468 1345 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:12.205307 1350 tablet_bootstrap.cc:654] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:12.205922 1345 tablet_bootstrap.cc:654] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:12.213275 1350 tablet_bootstrap.cc:492] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:12.213541 1350 ts_tablet_manager.cc:1367] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240503 08:13:12.214360 1350 raft_consensus.cc:348] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:12.214648 1347 ts_tablet_manager.cc:1398] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 08:13:12.214919 1350 raft_consensus.cc:374] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:12.215067 1350 raft_consensus.cc:729] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:12.215407 1350 consensus_queue.cc:260] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:12.216066 1350 ts_tablet_manager.cc:1398] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:12.217319 1345 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:12.217540 1345 ts_tablet_manager.cc:1367] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240503 08:13:12.218353 1347 tablet_bootstrap.cc:492] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:12.218361 1345 raft_consensus.cc:348] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:12.218664 1345 raft_consensus.cc:374] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:12.218772 1345 raft_consensus.cc:729] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:12.219120 1345 consensus_queue.cc:260] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:12.219956 1347 tablet_bootstrap.cc:654] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:12.220571 1345 ts_tablet_manager.cc:1398] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:12.224717 1344 tablet_bootstrap.cc:492] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:12.224881 1347 tablet_bootstrap.cc:492] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:12.224885 1344 ts_tablet_manager.cc:1367] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.012s
I20240503 08:13:12.225031 1347 ts_tablet_manager.cc:1367] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:13:12.225620 1347 raft_consensus.cc:348] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:12.225795 1347 raft_consensus.cc:374] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:12.225685 1344 raft_consensus.cc:348] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:12.225888 1344 raft_consensus.cc:374] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:12.225903 1347 raft_consensus.cc:729] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:12.225989 1344 raft_consensus.cc:729] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:12.226157 1347 consensus_queue.cc:260] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:12.226213 1344 consensus_queue.cc:260] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:12.226699 1347 ts_tablet_manager.cc:1398] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:12.226719 1344 ts_tablet_manager.cc:1398] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:12.227625 1287 raft_consensus.cc:482] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:12.227747 1287 raft_consensus.cc:504] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:12.228413 1287 leader_election.cc:288] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:12.229035 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9151dcb76174b27991fdcf3839db0a5" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:12.229010 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9151dcb76174b27991fdcf3839db0a5" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:12.229225 32532 raft_consensus.cc:2454] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:12.229230 32140 raft_consensus.cc:2454] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:12.229681 32206 leader_election.cc:302] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:13:12.230139 1287 raft_consensus.cc:2781] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:12.230250 1287 raft_consensus.cc:482] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:12.230337 1287 raft_consensus.cc:3037] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:12.232774 1287 raft_consensus.cc:504] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:12.233259 1287 leader_election.cc:288] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:12.233615 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9151dcb76174b27991fdcf3839db0a5" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:12.233698 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9151dcb76174b27991fdcf3839db0a5" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:12.233829 32140 raft_consensus.cc:3037] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:12.233850 32532 raft_consensus.cc:3037] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:12.236521 32532 raft_consensus.cc:2454] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:12.236521 32140 raft_consensus.cc:2454] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:12.236908 32205 leader_election.cc:302] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:12.237234 1287 raft_consensus.cc:2781] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:12.237380 1287 raft_consensus.cc:686] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:12.237645 1287 consensus_queue.cc:237] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:12.240510 31991 catalog_manager.cc:5484] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:12.275421 1278 raft_consensus.cc:482] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:12.275604 1278 raft_consensus.cc:504] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:12.276324 1278 leader_election.cc:288] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:12.276824 32272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf32f9e1e914653a6594d58c67cd80e" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:12.276922 32402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf32f9e1e914653a6594d58c67cd80e" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:12.277070 32272 raft_consensus.cc:2454] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:12.277141 32402 raft_consensus.cc:2454] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:12.277479 32466 leader_election.cc:302] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb; no voters:
I20240503 08:13:12.277765 1278 raft_consensus.cc:2781] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:12.277863 1278 raft_consensus.cc:482] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:12.277925 1278 raft_consensus.cc:3037] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:12.279781 1278 raft_consensus.cc:504] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:12.280287 1278 leader_election.cc:288] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:12.280764 32272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf32f9e1e914653a6594d58c67cd80e" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:12.280781 32402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf32f9e1e914653a6594d58c67cd80e" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:12.280964 32402 raft_consensus.cc:3037] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:12.280980 32272 raft_consensus.cc:3037] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:12.283635 32272 raft_consensus.cc:2454] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:12.283650 32402 raft_consensus.cc:2454] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:12.284044 32465 leader_election.cc:302] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:12.284340 1278 raft_consensus.cc:2781] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:12.284468 1278 raft_consensus.cc:686] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Becoming Leader. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Running, Role: LEADER
I20240503 08:13:12.284757 1278 consensus_queue.cc:237] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:12.287882 31991 catalog_manager.cc:5484] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:12.366479 1287 raft_consensus.cc:482] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:12.366673 1287 raft_consensus.cc:504] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:12.367287 1287 leader_election.cc:288] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:12.367743 32402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5c3fc4dc146d4818d6fdfbd4ac695" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:12.367794 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5c3fc4dc146d4818d6fdfbd4ac695" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:12.367990 32402 raft_consensus.cc:2454] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:12.368000 32140 raft_consensus.cc:2454] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 0.
I20240503 08:13:12.368402 32205 leader_election.cc:302] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:12.368662 1287 raft_consensus.cc:2781] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:12.368760 1287 raft_consensus.cc:482] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:12.368880 1287 raft_consensus.cc:3037] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:12.371017 1287 raft_consensus.cc:504] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:12.371668 1287 leader_election.cc:288] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Requested vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:12.371922 32402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5c3fc4dc146d4818d6fdfbd4ac695" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:12.372112 32402 raft_consensus.cc:3037] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:12.372167 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5c3fc4dc146d4818d6fdfbd4ac695" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:12.372361 32140 raft_consensus.cc:3037] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:12.373942 32402 raft_consensus.cc:2454] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:12.374262 32140 raft_consensus.cc:2454] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 1.
I20240503 08:13:12.374279 32205 leader_election.cc:302] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:12.374536 1287 raft_consensus.cc:2781] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:12.374688 1287 raft_consensus.cc:686] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:12.375006 1287 consensus_queue.cc:237] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } }
I20240503 08:13:12.378104 31991 catalog_manager.cc:5484] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:12.623450 1287 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:12.630064 1242 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:12.725658 1278 consensus_queue.cc:1035] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:12.727761 1360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:12.728168 1360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:13:12.730125 1197 consensus_queue.cc:1035] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:12.738745 1360 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 08:13:12.823891 1287 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:12.829808 1242 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:12.985394 32205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:41731, user_credentials={real_user=slave}} blocked reactor thread for 71671.2us
INSERT report
rows total: 2000
time total: 263.159 ms
time per row: 0.13158 ms
I20240503 08:13:13.165688 31993 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48936:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:13.174336 32120 tablet_service.cc:1467] Processing CreateTablet for tablet 95491576997c4392b482b4627760d818 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:13.174382 32381 tablet_service.cc:1467] Processing CreateTablet for tablet 95491576997c4392b482b4627760d818 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:13.174814 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95491576997c4392b482b4627760d818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.174825 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95491576997c4392b482b4627760d818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.174896 32121 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4ac5dea8024445ad2cd1ca724f2290 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:13.175220 32121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4ac5dea8024445ad2cd1ca724f2290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.176215 32510 tablet_service.cc:1467] Processing CreateTablet for tablet 10646e4785b047d8a3ac6af5ddb06474 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:13.176587 32510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10646e4785b047d8a3ac6af5ddb06474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.177827 32512 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4ac5dea8024445ad2cd1ca724f2290 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:13.178174 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4ac5dea8024445ad2cd1ca724f2290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.179468 32511 tablet_service.cc:1467] Processing CreateTablet for tablet 95491576997c4392b482b4627760d818 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:13.179838 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95491576997c4392b482b4627760d818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.183086 32251 tablet_service.cc:1467] Processing CreateTablet for tablet 10646e4785b047d8a3ac6af5ddb06474 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:13.183478 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10646e4785b047d8a3ac6af5ddb06474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.188166 32380 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4ac5dea8024445ad2cd1ca724f2290 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:13.188393 1427 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:13.188581 32380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4ac5dea8024445ad2cd1ca724f2290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.188786 1425 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:13.189567 32119 tablet_service.cc:1467] Processing CreateTablet for tablet 10646e4785b047d8a3ac6af5ddb06474 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:13.189937 32119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10646e4785b047d8a3ac6af5ddb06474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:13.190137 1427 tablet_bootstrap.cc:654] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:13.190443 1425 tablet_bootstrap.cc:654] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:13.204455 1427 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:13.204700 1427 ts_tablet_manager.cc:1367] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.004s
I20240503 08:13:13.205577 1427 raft_consensus.cc:348] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.205843 1427 raft_consensus.cc:374] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:13.205965 1427 raft_consensus.cc:729] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.206233 1427 consensus_queue.cc:260] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.207633 1430 tablet_bootstrap.cc:492] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb: Bootstrap starting.
I20240503 08:13:13.208011 1427 ts_tablet_manager.cc:1398] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:13.208328 1427 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79: Bootstrap starting.
I20240503 08:13:13.211107 1427 tablet_bootstrap.cc:654] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:13.211383 1430 tablet_bootstrap.cc:654] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:13.214468 1426 tablet_bootstrap.cc:492] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:13.215831 1427 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79: No bootstrap required, opened a new log
I20240503 08:13:13.216037 1427 ts_tablet_manager.cc:1367] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240503 08:13:13.216720 1427 raft_consensus.cc:348] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:13.216854 1426 tablet_bootstrap.cc:654] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:13.216935 1427 raft_consensus.cc:374] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:13.217006 1427 raft_consensus.cc:729] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.217013 1425 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:13.217262 1427 consensus_queue.cc:260] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:13.217149 1425 ts_tablet_manager.cc:1367] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.010s
I20240503 08:13:13.217885 1427 ts_tablet_manager.cc:1398] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:13.218487 1425 raft_consensus.cc:348] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.218694 1425 raft_consensus.cc:374] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:13.218827 1425 raft_consensus.cc:729] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.219094 1425 consensus_queue.cc:260] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.219552 1430 tablet_bootstrap.cc:492] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb: No bootstrap required, opened a new log
I20240503 08:13:13.219698 1430 ts_tablet_manager.cc:1367] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 08:13:13.220626 1425 ts_tablet_manager.cc:1398] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:13.220935 1425 tablet_bootstrap.cc:492] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:13.221136 1430 raft_consensus.cc:348] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.221339 1430 raft_consensus.cc:374] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:13.221446 1430 raft_consensus.cc:729] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.221722 1430 consensus_queue.cc:260] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.222438 1430 ts_tablet_manager.cc:1398] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:13.226373 1425 tablet_bootstrap.cc:654] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:13.226778 1426 tablet_bootstrap.cc:492] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:13.226958 1426 ts_tablet_manager.cc:1367] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240503 08:13:13.227581 1426 raft_consensus.cc:348] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.227725 1426 raft_consensus.cc:374] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:13.227797 1426 raft_consensus.cc:729] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.228014 1426 consensus_queue.cc:260] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.228171 1425 tablet_bootstrap.cc:492] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:13.228340 1425 ts_tablet_manager.cc:1367] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:13:13.228511 1426 ts_tablet_manager.cc:1398] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:13.228864 1426 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:13.229027 1425 raft_consensus.cc:348] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.229501 1425 raft_consensus.cc:374] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:13.229621 1425 raft_consensus.cc:729] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.229909 1425 consensus_queue.cc:260] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.230454 1426 tablet_bootstrap.cc:654] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:13.230849 1425 ts_tablet_manager.cc:1398] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:13.231356 1425 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90: Bootstrap starting.
I20240503 08:13:13.232972 1425 tablet_bootstrap.cc:654] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:13.235841 1425 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90: No bootstrap required, opened a new log
I20240503 08:13:13.235970 1425 ts_tablet_manager.cc:1367] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 08:13:13.236218 1426 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:13.236364 1426 ts_tablet_manager.cc:1367] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240503 08:13:13.236647 1425 raft_consensus.cc:348] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:13.236832 1425 raft_consensus.cc:374] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:13.236927 1425 raft_consensus.cc:729] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.237102 1426 raft_consensus.cc:348] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:13.237133 1425 consensus_queue.cc:260] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:13.237308 1426 raft_consensus.cc:374] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:13.237390 1426 raft_consensus.cc:729] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.237633 1425 ts_tablet_manager.cc:1398] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:13.237610 1426 consensus_queue.cc:260] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:13.238034 1426 ts_tablet_manager.cc:1398] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:13.238308 1426 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797: Bootstrap starting.
I20240503 08:13:13.239477 1426 tablet_bootstrap.cc:654] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:13.241019 1426 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797: No bootstrap required, opened a new log
I20240503 08:13:13.241148 1426 ts_tablet_manager.cc:1367] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:13.241779 1426 raft_consensus.cc:348] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.241963 1426 raft_consensus.cc:374] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:13.242059 1426 raft_consensus.cc:729] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Initialized, Role: FOLLOWER
I20240503 08:13:13.242328 1426 consensus_queue.cc:260] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.242803 1426 ts_tablet_manager.cc:1398] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:13.268512 1417 raft_consensus.cc:482] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:13.268687 1417 raft_consensus.cc:504] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:13.269327 1417 leader_election.cc:288] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:13.269677 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4ac5dea8024445ad2cd1ca724f2290" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:13.269743 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4ac5dea8024445ad2cd1ca724f2290" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79" is_pre_election: true
I20240503 08:13:13.269901 32141 raft_consensus.cc:2454] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:13.269977 32400 raft_consensus.cc:2454] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 0.
I20240503 08:13:13.270236 32465 leader_election.cc:302] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:13.270480 1417 raft_consensus.cc:2781] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:13.270574 1417 raft_consensus.cc:482] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:13.270658 1417 raft_consensus.cc:3037] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:13.272507 1417 raft_consensus.cc:504] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:13.272984 1417 leader_election.cc:288] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:13.273418 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4ac5dea8024445ad2cd1ca724f2290" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:13.273418 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4ac5dea8024445ad2cd1ca724f2290" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:13.273630 32141 raft_consensus.cc:3037] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:13.273629 32400 raft_consensus.cc:3037] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:13.276327 32141 raft_consensus.cc:2454] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:13.276327 32400 raft_consensus.cc:2454] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 1.
I20240503 08:13:13.276705 32465 leader_election.cc:302] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:13.276953 1417 raft_consensus.cc:2781] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:13.277089 1417 raft_consensus.cc:686] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Becoming Leader. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Running, Role: LEADER
I20240503 08:13:13.277362 1417 consensus_queue.cc:237] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } }
I20240503 08:13:13.280673 31993 catalog_manager.cc:5484] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:13.363302 1069 raft_consensus.cc:482] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:13.363508 1069 raft_consensus.cc:504] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.364063 1069 leader_election.cc:288] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:13.364459 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95491576997c4392b482b4627760d818" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" is_pre_election: true
I20240503 08:13:13.364532 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95491576997c4392b482b4627760d818" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:13.364646 32141 raft_consensus.cc:2454] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 0.
I20240503 08:13:13.364745 32532 raft_consensus.cc:2454] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 0.
I20240503 08:13:13.364977 32335 leader_election.cc:302] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:13.365222 1069 raft_consensus.cc:2781] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:13.365312 1069 raft_consensus.cc:482] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:13.365406 1069 raft_consensus.cc:3037] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:13.367269 1069 raft_consensus.cc:504] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.368105 1069 leader_election.cc:288] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:13.368320 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95491576997c4392b482b4627760d818" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:13.368325 32138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95491576997c4392b482b4627760d818" candidate_uuid: "71ad6bf07682497aaa1e9429320aef79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:13.368501 32138 raft_consensus.cc:3037] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:13.368510 32532 raft_consensus.cc:3037] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:13.371119 32532 raft_consensus.cc:2454] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 1.
I20240503 08:13:13.371133 32138 raft_consensus.cc:2454] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ad6bf07682497aaa1e9429320aef79 in term 1.
I20240503 08:13:13.371502 32336 leader_election.cc:302] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:13.371729 1069 raft_consensus.cc:2781] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:13.372098 1069 raft_consensus.cc:686] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Becoming Leader. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Running, Role: LEADER
I20240503 08:13:13.372347 1069 consensus_queue.cc:237] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.375352 31993 catalog_manager.cc:5484] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:13.491794 1286 raft_consensus.cc:482] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:13.491961 1286 raft_consensus.cc:504] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.492566 1286 leader_election.cc:288] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:13.492942 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10646e4785b047d8a3ac6af5ddb06474" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb" is_pre_election: true
I20240503 08:13:13.493144 32271 raft_consensus.cc:2454] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:13.493124 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10646e4785b047d8a3ac6af5ddb06474" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797" is_pre_election: true
I20240503 08:13:13.493360 32532 raft_consensus.cc:2454] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 0.
I20240503 08:13:13.493544 32075 leader_election.cc:302] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:13.493788 1286 raft_consensus.cc:2781] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:13.493886 1286 raft_consensus.cc:482] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:13.493964 1286 raft_consensus.cc:3037] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:13.496181 1286 raft_consensus.cc:504] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.496706 1286 leader_election.cc:288] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:13.497041 32271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10646e4785b047d8a3ac6af5ddb06474" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:13.497138 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10646e4785b047d8a3ac6af5ddb06474" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:13.497228 32271 raft_consensus.cc:3037] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:13.497319 32532 raft_consensus.cc:3037] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:13.499897 32271 raft_consensus.cc:2454] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:13.499897 32532 raft_consensus.cc:2454] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 1.
I20240503 08:13:13.500276 32075 leader_election.cc:302] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:13.500525 1286 raft_consensus.cc:2781] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:13.500844 1286 raft_consensus.cc:686] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Leader. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:13.501116 1286 consensus_queue.cc:237] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } }
I20240503 08:13:13.504056 31993 catalog_manager.cc:5484] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:13.674364 1417 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:13.678642 1417 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:13.788919 1069 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:13.794176 1069 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:13.878845 1286 consensus_queue.cc:1035] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:13.883946 1286 consensus_queue.cc:1035] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:13.905164 1444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:13.905495 1444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:13.913733 1444 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 207.8 ms
time per row: 0.1039 ms
I20240503 08:13:14.260438 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.5:44035
--local_ip_for_outbound_sockets=127.24.123.5
--tserver_master_addrs=127.24.123.62:44195
--webserver_port=45317
--webserver_interface=127.24.123.5
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 08:13:14.594702 1508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:14.595052 1508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:14.595218 1508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:14.603847 1508 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:14.604003 1508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:14.604140 1508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.5
I20240503 08:13:14.613384 1508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.5:44035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.123.5
--webserver_port=45317
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.5
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:14.614606 1508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:14.616319 1508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:14.627672 1514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:14.628072 1513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:14.628544 1516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:14.629065 1508 server_base.cc:1034] running on GCE node
I20240503 08:13:14.629644 1508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:14.630457 1508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:14.631651 1508 hybrid_clock.cc:648] HybridClock initialized: now 1714723994631602 us; error 41 us; skew 500 ppm
I20240503 08:13:14.631969 1508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:14.634533 1508 webserver.cc:460] Webserver started at http://127.24.123.5:45317/ using document root <none> and password file <none>
I20240503 08:13:14.635301 1508 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:14.635439 1508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:14.640908 1508 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:13:14.643646 1522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:14.645035 1508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:14.645179 1508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c6b280e8ecd54855b7b21b58a223149a"
format_stamp: "Formatted at 2024-05-03 08:12:53 on dist-test-slave-gkrw"
I20240503 08:13:14.645970 1508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:14.672044 1508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:14.672806 1508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:14.673066 1508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:14.673923 1508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:14.675415 1508 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:14.675489 1508 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:14.675602 1508 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:14.675652 1508 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:14.705466 1508 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.5:44035
I20240503 08:13:14.705518 1635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.5:44035 every 8 connection(s)
I20240503 08:13:14.706790 1508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:13:14.715602 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 1508
I20240503 08:13:14.716302 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.6:36447
--local_ip_for_outbound_sockets=127.24.123.6
--tserver_master_addrs=127.24.123.62:44195
--webserver_port=32843
--webserver_interface=127.24.123.6
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:13:14.719022 1636 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:13:14.719339 1636 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:14.720238 1636 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:13:14.721781 31995 ts_manager.cc:194] Re-registered known tserver with Master: c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035)
I20240503 08:13:14.722602 31995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.5:52665
W20240503 08:13:15.053110 1640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:15.053486 1640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:15.053632 1640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:15.062753 1640 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:15.062916 1640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:15.063098 1640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.6
I20240503 08:13:15.072247 1640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.6:36447
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.123.6
--webserver_port=32843
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.6
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:15.073392 1640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:15.074946 1640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:15.086937 1648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:15.087622 1647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:15.087795 1650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:15.092429 1640 server_base.cc:1034] running on GCE node
I20240503 08:13:15.092993 1640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:15.093920 1640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:15.095129 1640 hybrid_clock.cc:648] HybridClock initialized: now 1714723995095057 us; error 56 us; skew 500 ppm
I20240503 08:13:15.095479 1640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:15.097982 1640 webserver.cc:460] Webserver started at http://127.24.123.6:32843/ using document root <none> and password file <none>
I20240503 08:13:15.098919 1640 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:15.099056 1640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:15.104705 1640 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:13:15.107493 1656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:15.108897 1640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 08:13:15.109035 1640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c93e47973e047d0a162a50f72f9a757"
format_stamp: "Formatted at 2024-05-03 08:12:53 on dist-test-slave-gkrw"
I20240503 08:13:15.109797 1640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:15.150389 1640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:15.151273 1640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:15.151553 1640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:15.152447 1640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:15.153889 1640 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:15.153968 1640 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:15.154093 1640 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:15.154145 1640 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:15.187944 1640 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.6:36447
I20240503 08:13:15.187991 1769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.6:36447 every 8 connection(s)
I20240503 08:13:15.189311 1640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:13:15.195102 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 1640
I20240503 08:13:15.195770 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.7:33603
--local_ip_for_outbound_sockets=127.24.123.7
--tserver_master_addrs=127.24.123.62:44195
--webserver_port=33787
--webserver_interface=127.24.123.7
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:13:15.201671 1770 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:13:15.201973 1770 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:15.202920 1770 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:13:15.204360 31995 ts_manager.cc:194] Re-registered known tserver with Master: 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447)
I20240503 08:13:15.205279 31995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.6:37357
W20240503 08:13:15.525663 1773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:15.525931 1773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:15.526034 1773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:15.533468 1773 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:15.533579 1773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:15.533700 1773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.7
I20240503 08:13:15.546768 1773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.7:33603
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.123.7
--webserver_port=33787
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.7
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:15.548384 1773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:15.550532 1773 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:15.563773 1779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:15.564116 1780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:15.564430 1782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:15.565032 1773 server_base.cc:1034] running on GCE node
I20240503 08:13:15.565562 1773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:15.566386 1773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:15.567585 1773 hybrid_clock.cc:648] HybridClock initialized: now 1714723995567530 us; error 47 us; skew 500 ppm
I20240503 08:13:15.567911 1773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:15.570040 1773 webserver.cc:460] Webserver started at http://127.24.123.7:33787/ using document root <none> and password file <none>
I20240503 08:13:15.570703 1773 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:15.570820 1773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:15.575839 1773 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:13:15.578277 1788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:15.579545 1773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:13:15.579682 1773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f7f6592d13d245508261964ca6ffc5c4"
format_stamp: "Formatted at 2024-05-03 08:12:54 on dist-test-slave-gkrw"
I20240503 08:13:15.580431 1773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:15.597183 1773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:15.597929 1773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:15.598178 1773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:15.599066 1773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:15.600533 1773 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:15.600605 1773 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:15.600747 1773 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:15.600824 1773 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:15.633394 1773 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.7:33603
I20240503 08:13:15.633400 1901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.7:33603 every 8 connection(s)
I20240503 08:13:15.635136 1773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:13:15.644649 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 1773
I20240503 08:13:15.645355 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.8:43749
--local_ip_for_outbound_sockets=127.24.123.8
--tserver_master_addrs=127.24.123.62:44195
--webserver_port=35997
--webserver_interface=127.24.123.8
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:13:15.649379 1902 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:13:15.649755 1902 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:15.650555 1902 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:13:15.651968 31995 ts_manager.cc:194] Re-registered known tserver with Master: f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603)
I20240503 08:13:15.652730 31995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.7:55183
I20240503 08:13:15.725291 1636 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
W20240503 08:13:15.981398 1905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:15.981755 1905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:15.981905 1905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:15.989738 1905 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:15.989847 1905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:15.989995 1905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.8
I20240503 08:13:15.999281 1905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.8:43749
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.123.8
--webserver_port=35997
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.8
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:16.000494 1905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:16.002092 1905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:16.013551 1911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:16.013902 1914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:16.014492 1912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:16.014966 1905 server_base.cc:1034] running on GCE node
I20240503 08:13:16.015589 1905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:16.016466 1905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:16.017655 1905 hybrid_clock.cc:648] HybridClock initialized: now 1714723996017609 us; error 43 us; skew 500 ppm
I20240503 08:13:16.017944 1905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:16.020002 1905 webserver.cc:460] Webserver started at http://127.24.123.8:35997/ using document root <none> and password file <none>
I20240503 08:13:16.020675 1905 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:16.020812 1905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:16.025905 1905 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:13:16.028405 1920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:16.029625 1905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:16.029762 1905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e615bd1588964e2fbe14f14f4aae0a98"
format_stamp: "Formatted at 2024-05-03 08:12:54 on dist-test-slave-gkrw"
I20240503 08:13:16.030498 1905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:16.055676 1905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:16.056663 1905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:16.057013 1905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:16.058033 1905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:16.059489 1905 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:16.059561 1905 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:16.059674 1905 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:16.059722 1905 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:16.093227 1905 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.8:43749
I20240503 08:13:16.093264 2033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.8:43749 every 8 connection(s)
I20240503 08:13:16.094650 1905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:13:16.096956 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 1905
I20240503 08:13:16.108315 2034 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:13:16.108613 2034 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:16.109982 2034 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:13:16.111637 31995 ts_manager.cc:194] Re-registered known tserver with Master: e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749)
I20240503 08:13:16.112558 31995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.8:56315
I20240503 08:13:16.207784 1770 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
W20240503 08:13:16.449168 2040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:16.449455 2040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:16.457026 2040 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:16.655030 1902 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
I20240503 08:13:16.703073 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:16.719009 32251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:16.736765 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:16.748778 1705 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.757975 32251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:16.759663 1705 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.779114 32380 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:16.780364 32380 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:16.828385 32119 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:16.834461 1571 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.849599 1837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.860023 32119 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:16.877602 1969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.911104 1969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.920197 1571 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:16.921162 1837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.117084 2034 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
I20240503 08:13:17.215934 1705 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.216970 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:17.219594 32251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:17.253072 32380 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:17.275398 32119 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:17.294636 1571 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.338018 1837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.338960 1969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.617115 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:17.626320 32251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:17.630227 32380 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:17.639336 1705 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.650626 32119 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:17.662552 1571 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.687242 1969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:17.689529 1837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:13:17.726171 2171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:17.726670 2171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:17.738409 2171 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:17.815722 32272 consensus_queue.cc:237] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:17.819661 32141 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:17.819661 32400 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:17.820277 1906 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:17.821359 1906 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:17.824806 2042 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:17.825480 32400 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:17.825706 32141 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:17.831329 32000 catalog_manager.cc:5484] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:17.843215 32207 consensus_peers.cc:485] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27a469ffc17447ed86860e991ff3534a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:17.856218 32532 consensus_queue.cc:237] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:17.868607 32141 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:17.868678 32400 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:17.869693 2259 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:17.871882 2259 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:17.872942 2041 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:17.884155 31993 catalog_manager.cc:5484] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:17.884187 32400 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
W20240503 08:13:17.904788 32465 consensus_peers.cc:485] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f4ac5dea8024445ad2cd1ca724f2290. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:17.907116 32141 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:17.931727 32400 consensus_queue.cc:237] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:17.951184 32532 raft_consensus.cc:1261] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:17.951812 2121 consensus_queue.cc:1035] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:17.955523 32141 raft_consensus.cc:1261] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:17.956120 2121 consensus_queue.cc:1035] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:17.960161 32336 consensus_peers.cc:485] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6b5cbb6708049e6ae32da12bb27368f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:17.969012 2121 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:17.974051 32141 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:17.975666 32000 catalog_manager.cc:5484] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:17.978089 32532 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.010138 32251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:18.012428 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:18.025197 1705 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.033447 32380 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:18.059468 32272 consensus_queue.cc:237] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:18.062031 1571 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.074148 32119 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:18.083225 32141 raft_consensus.cc:1261] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.083945 2242 consensus_queue.cc:1035] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.086587 32532 raft_consensus.cc:1261] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.087167 2242 consensus_queue.cc:1035] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:18.088111 32207 consensus_peers.cc:485] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afdc26bd51bd467f88d7892edb81ca9a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.094161 2242 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.104564 32532 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.110392 32000 catalog_manager.cc:5484] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.134191 32138 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.135486 1969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.174409 1837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:18.228816 32532 consensus_queue.cc:237] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:18.231731 32272 raft_consensus.cc:1261] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.232013 32400 raft_consensus.cc:1261] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.232307 2041 consensus_queue.cc:1035] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.232878 2041 consensus_queue.cc:1035] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:18.238977 32467 consensus_peers.cc:485] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cf32f9e1e914653a6594d58c67cd80e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.240877 2041 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.241607 32400 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.249032 32272 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.249770 31995 catalog_manager.cc:5484] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
Runtime error: ksck discovered errors
I20240503 08:13:18.268180 32272 consensus_queue.cc:237] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:18.270519 32400 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.271245 1906 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.272055 32141 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.272532 1906 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.275110 1906 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
W20240503 08:13:18.277855 32206 consensus_peers.cc:485] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9541e40695844a1b05ca1bc0dd8a10d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.278434 32400 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.281915 32141 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.284348 31995 catalog_manager.cc:5484] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.304443 32141 consensus_queue.cc:237] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:18.307431 32272 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.308068 1774 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.315384 32532 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.315976 1774 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.315868 2328 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
W20240503 08:13:18.327234 32074 consensus_peers.cc:485] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bd736780ea840ab9b6630a4acd80ae9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.332008 32272 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:18.332876 31994 catalog_manager.cc:5484] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:18.336764 32271 consensus_queue.cc:237] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:18.339221 32532 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:18.344077 32141 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:18.344298 32207 consensus_peers.cc:485] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25f1956062e84b92a3b036fc5dff4b46. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.344648 1906 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.345994 32532 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.346508 1906 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.347373 2242 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.349637 32532 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.352890 31994 catalog_manager.cc:5484] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:18.353832 32141 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.378443 32532 consensus_queue.cc:237] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:18.380962 32141 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.381350 32271 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.381940 2041 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.382407 2041 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.385481 2041 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
W20240503 08:13:18.387499 32466 consensus_peers.cc:485] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 734d2de8d2804b2c8d2f144157e12f77. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.389153 32271 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.389233 32141 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.393697 32000 catalog_manager.cc:5484] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.397445 2350 ts_tablet_manager.cc:902] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:18.399194 2350 tablet_copy_client.cc:308] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:18.407315 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet 2f4ac5dea8024445ad2cd1ca724f2290 from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:35625)
I20240503 08:13:18.407547 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet 2f4ac5dea8024445ad2cd1ca724f2290 from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:35625: session id = c6b280e8ecd54855b7b21b58a223149a-2f4ac5dea8024445ad2cd1ca724f2290
I20240503 08:13:18.410660 32552 tablet_copy_source_session.cc:205] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:18.415450 2350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4ac5dea8024445ad2cd1ca724f2290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:18.420652 2361 ts_tablet_manager.cc:902] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:18.423893 32141 consensus_queue.cc:237] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:18.426985 32271 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.427567 32400 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.427598 2328 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.428371 2350 tablet_copy_client.cc:783] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:18.428346 2328 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.429023 2350 tablet_copy_client.cc:647] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:18.432806 2328 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.435020 2354 tablet_copy_client.cc:1002] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
W20240503 08:13:18.435647 32076 consensus_peers.cc:485] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cc8a3241af248578b6ce87b1e33cea9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.437110 32271 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.439818 31994 catalog_manager.cc:5484] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:18.449606 2361 tablet_copy_client.cc:308] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:18.454968 32400 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.464828 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet 27a469ffc17447ed86860e991ff3534a from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:35733)
I20240503 08:13:18.465134 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet 27a469ffc17447ed86860e991ff3534a from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:35733: session id = e615bd1588964e2fbe14f14f4aae0a98-27a469ffc17447ed86860e991ff3534a
I20240503 08:13:18.469040 32292 tablet_copy_source_session.cc:205] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:18.474220 2361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a469ffc17447ed86860e991ff3534a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:18.474936 32271 consensus_queue.cc:237] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:18.477625 32141 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.477816 32532 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.478417 2042 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.480417 2042 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.481943 2042 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
W20240503 08:13:18.486894 32205 consensus_peers.cc:485] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5de086cc1f0b497f87b055e745a38e0d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.487267 32532 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:18.492791 31993 catalog_manager.cc:5484] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:18.496441 32141 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:18.506850 2361 tablet_copy_client.cc:783] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:18.507203 2361 tablet_copy_client.cc:647] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:18.509559 2370 tablet_copy_client.cc:1002] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:18.510263 2384 ts_tablet_manager.cc:902] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:18.515780 32400 consensus_queue.cc:237] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:18.518121 32141 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:18.518579 32336 consensus_peers.cc:485] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95491576997c4392b482b4627760d818. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.520566 2121 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.523640 2121 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.525992 32141 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.531056 32530 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.531450 31993 catalog_manager.cc:5484] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.542061 2391 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.560910 32530 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.575400 2384 tablet_copy_client.cc:308] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:34237
I20240503 08:13:18.604454 2403 ts_tablet_manager.cc:902] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:18.604511 32422 tablet_copy_service.cc:140] P 71ad6bf07682497aaa1e9429320aef79: Received BeginTabletCopySession request for tablet f6b5cbb6708049e6ae32da12bb27368f from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:47045)
I20240503 08:13:18.604768 32422 tablet_copy_service.cc:161] P 71ad6bf07682497aaa1e9429320aef79: Beginning new tablet copy session on tablet f6b5cbb6708049e6ae32da12bb27368f from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:47045: session id = 6c93e47973e047d0a162a50f72f9a757-f6b5cbb6708049e6ae32da12bb27368f
I20240503 08:13:18.608191 32422 tablet_copy_source_session.cc:205] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:18.609269 32400 consensus_queue.cc:237] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:18.615382 2384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b5cbb6708049e6ae32da12bb27368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:18.616037 32271 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.616679 2391 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.616937 32141 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.617511 2390 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.621743 2390 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.627323 32271 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.627817 31993 catalog_manager.cc:5484] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:18.629932 32141 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.645423 2384 tablet_copy_client.cc:783] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:18.645866 2384 tablet_copy_client.cc:647] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:18.649577 2399 tablet_copy_client.cc:1002] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
W20240503 08:13:18.654209 32337 consensus_peers.cc:485] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af5c3865ca694bf090dce19f1100e5eb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.654656 2403 tablet_copy_client.cc:308] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:18.674803 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet afdc26bd51bd467f88d7892edb81ca9a from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:37127)
I20240503 08:13:18.675031 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet afdc26bd51bd467f88d7892edb81ca9a from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:37127: session id = f7f6592d13d245508261964ca6ffc5c4-afdc26bd51bd467f88d7892edb81ca9a
I20240503 08:13:18.678022 32292 tablet_copy_source_session.cc:205] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:18.683585 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc26bd51bd467f88d7892edb81ca9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:18.699169 2403 tablet_copy_client.cc:783] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:18.699590 2403 tablet_copy_client.cc:647] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:18.700189 2417 ts_tablet_manager.cc:902] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:18.701478 2411 tablet_copy_client.cc:1002] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:18.715646 2417 tablet_copy_client.cc:308] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:18.717434 32141 consensus_queue.cc:237] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:18.723435 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet 2cf32f9e1e914653a6594d58c67cd80e from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:48191)
I20240503 08:13:18.723632 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet 2cf32f9e1e914653a6594d58c67cd80e from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:48191: session id = e615bd1588964e2fbe14f14f4aae0a98-2cf32f9e1e914653a6594d58c67cd80e
I20240503 08:13:18.726697 32552 tablet_copy_source_session.cc:205] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:18.727903 32530 raft_consensus.cc:1261] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.728520 32400 raft_consensus.cc:1261] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:18.728667 32076 consensus_peers.cc:485] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7996d398e3440ebbb8f942bda8f6833. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.729097 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf32f9e1e914653a6594d58c67cd80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:18.729146 2331 consensus_queue.cc:1035] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.729497 2328 consensus_queue.cc:1035] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.735647 2417 tablet_copy_client.cc:783] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:18.736007 2417 tablet_copy_client.cc:647] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:18.737152 1774 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.738010 32400 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.739627 32530 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.740772 2422 tablet_copy_client.cc:1002] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:18.742414 31994 catalog_manager.cc:5484] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:18.762555 32271 consensus_queue.cc:237] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:18.765244 32530 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.765471 32141 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.765765 1906 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:18.766069 32205 consensus_peers.cc:485] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9151dcb76174b27991fdcf3839db0a5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.766247 1906 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.768712 2042 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:18.771211 32530 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:18.778357 31994 catalog_manager.cc:5484] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:18.784659 32138 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:18.795790 2439 ts_tablet_manager.cc:902] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:18.798435 2439 tablet_copy_client.cc:308] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:18.802958 32271 consensus_queue.cc:237] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:18.807787 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet f9541e40695844a1b05ca1bc0dd8a10d from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:41801)
I20240503 08:13:18.807934 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet f9541e40695844a1b05ca1bc0dd8a10d from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:41801: session id = 6c93e47973e047d0a162a50f72f9a757-f9541e40695844a1b05ca1bc0dd8a10d
W20240503 08:13:18.808172 32207 consensus_peers.cc:485] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 400a0317f9184c99a449f7f160fa998f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.808434 32138 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.809613 2042 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.810328 32400 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.810609 32292 tablet_copy_source_session.cc:205] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:18.811327 2042 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.819173 2439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9541e40695844a1b05ca1bc0dd8a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:18.819905 2042 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.820575 32141 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.826589 31993 catalog_manager.cc:5484] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.829564 32401 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:18.842782 2439 tablet_copy_client.cc:783] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:18.843044 2439 tablet_copy_client.cc:647] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:18.844648 2443 tablet_copy_client.cc:1002] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:18.850193 32271 consensus_queue.cc:237] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:18.852671 32400 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.852887 32530 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.853821 1906 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:18.854629 32206 consensus_peers.cc:485] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e6a81a3850e49edb08c8060a45b6831. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.855569 1906 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.857132 1906 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.859433 32400 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.863332 2453 ts_tablet_manager.cc:902] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:18.867396 31994 catalog_manager.cc:5484] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.871577 2457 ts_tablet_manager.cc:902] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:18.873986 32532 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:18.893862 2453 tablet_copy_client.cc:308] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:18.904388 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet 2bd736780ea840ab9b6630a4acd80ae9 from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:56869)
I20240503 08:13:18.904635 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet 2bd736780ea840ab9b6630a4acd80ae9 from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:56869: session id = c6b280e8ecd54855b7b21b58a223149a-2bd736780ea840ab9b6630a4acd80ae9
I20240503 08:13:18.907287 32161 tablet_copy_source_session.cc:205] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:18.909462 2453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd736780ea840ab9b6630a4acd80ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:18.916404 2453 tablet_copy_client.cc:783] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:18.916755 2453 tablet_copy_client.cc:647] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:18.919050 2464 tablet_copy_client.cc:1002] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:18.919579 2457 tablet_copy_client.cc:308] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:18.925835 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet 734d2de8d2804b2c8d2f144157e12f77 from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:52495)
I20240503 08:13:18.925998 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet 734d2de8d2804b2c8d2f144157e12f77 from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:52495: session id = 6c93e47973e047d0a162a50f72f9a757-734d2de8d2804b2c8d2f144157e12f77
I20240503 08:13:18.928828 32552 tablet_copy_source_session.cc:205] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:18.930615 2457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734d2de8d2804b2c8d2f144157e12f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:18.935302 2457 tablet_copy_client.cc:783] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:18.935529 2457 tablet_copy_client.cc:647] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:18.937140 2467 tablet_copy_client.cc:1002] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:18.938786 32138 consensus_queue.cc:237] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:18.941259 32271 raft_consensus.cc:1261] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:18.941509 32530 raft_consensus.cc:1261] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:18.941854 32076 consensus_peers.cc:485] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10646e4785b047d8a3ac6af5ddb06474. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:18.942278 2328 consensus_queue.cc:1035] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.942792 2328 consensus_queue.cc:1035] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:18.955343 2331 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.959476 32532 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.967751 31994 catalog_manager.cc:5484] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:18.973454 2474 ts_tablet_manager.cc:902] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:19.021944 32271 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:19.026491 2474 tablet_copy_client.cc:308] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:19.026691 2485 ts_tablet_manager.cc:902] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:19.027324 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet 25f1956062e84b92a3b036fc5dff4b46 from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:37127)
I20240503 08:13:19.027485 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet 25f1956062e84b92a3b036fc5dff4b46 from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:37127: session id = f7f6592d13d245508261964ca6ffc5c4-25f1956062e84b92a3b036fc5dff4b46
I20240503 08:13:19.030441 32292 tablet_copy_source_session.cc:205] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.031842 2474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f1956062e84b92a3b036fc5dff4b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.051645 2493 ts_tablet_manager.cc:902] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:19.059888 2474 tablet_copy_client.cc:783] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.060209 2474 tablet_copy_client.cc:647] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.062350 2491 tablet_copy_client.cc:1002] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
W20240503 08:13:19.068881 2362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:19.069291 2362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:13:19.069993 2493 tablet_copy_client.cc:308] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:19.077497 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet 7cc8a3241af248578b6ce87b1e33cea9 from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:46063)
I20240503 08:13:19.077672 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet 7cc8a3241af248578b6ce87b1e33cea9 from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:46063: session id = e615bd1588964e2fbe14f14f4aae0a98-7cc8a3241af248578b6ce87b1e33cea9
W20240503 08:13:19.080842 2362 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:19.081138 32161 tablet_copy_source_session.cc:205] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.083293 2493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc8a3241af248578b6ce87b1e33cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.094089 2493 tablet_copy_client.cc:783] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.094331 2493 tablet_copy_client.cc:647] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.096511 2501 tablet_copy_client.cc:1002] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:19.101673 2511 ts_tablet_manager.cc:902] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:19.120846 2485 tablet_copy_client.cc:308] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:34237
I20240503 08:13:19.121789 32422 tablet_copy_service.cc:140] P 71ad6bf07682497aaa1e9429320aef79: Received BeginTabletCopySession request for tablet 95491576997c4392b482b4627760d818 from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:47045)
I20240503 08:13:19.122004 32422 tablet_copy_service.cc:161] P 71ad6bf07682497aaa1e9429320aef79: Beginning new tablet copy session on tablet 95491576997c4392b482b4627760d818 from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:47045: session id = 6c93e47973e047d0a162a50f72f9a757-95491576997c4392b482b4627760d818
I20240503 08:13:19.125097 32422 tablet_copy_source_session.cc:205] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.125326 2511 tablet_copy_client.cc:308] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:19.135304 2485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95491576997c4392b482b4627760d818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.135757 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet 5de086cc1f0b497f87b055e745a38e0d from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:49993)
I20240503 08:13:19.135913 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet 5de086cc1f0b497f87b055e745a38e0d from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:49993: session id = c6b280e8ecd54855b7b21b58a223149a-5de086cc1f0b497f87b055e745a38e0d
I20240503 08:13:19.138561 32292 tablet_copy_source_session.cc:205] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.139851 2511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de086cc1f0b497f87b055e745a38e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.147117 2511 tablet_copy_client.cc:783] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.147440 2511 tablet_copy_client.cc:647] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.148980 2518 tablet_copy_client.cc:1002] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:19.152956 2485 tablet_copy_client.cc:783] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.153306 2485 tablet_copy_client.cc:647] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.155735 2510 tablet_copy_client.cc:1002] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:19.225055 2539 ts_tablet_manager.cc:902] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:19.227591 2539 tablet_copy_client.cc:308] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:34237
I20240503 08:13:19.235014 32422 tablet_copy_service.cc:140] P 71ad6bf07682497aaa1e9429320aef79: Received BeginTabletCopySession request for tablet af5c3865ca694bf090dce19f1100e5eb from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:48777)
I20240503 08:13:19.235190 32422 tablet_copy_service.cc:161] P 71ad6bf07682497aaa1e9429320aef79: Beginning new tablet copy session on tablet af5c3865ca694bf090dce19f1100e5eb from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:48777: session id = f7f6592d13d245508261964ca6ffc5c4-af5c3865ca694bf090dce19f1100e5eb
I20240503 08:13:19.237414 32422 tablet_copy_source_session.cc:205] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.239231 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5c3865ca694bf090dce19f1100e5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.245781 2539 tablet_copy_client.cc:783] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.246091 2539 tablet_copy_client.cc:647] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.248056 2548 tablet_copy_client.cc:1002] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:19.285133 2590 ts_tablet_manager.cc:902] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:19.292976 2590 tablet_copy_client.cc:308] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:19.293658 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet 400a0317f9184c99a449f7f160fa998f from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:37127)
I20240503 08:13:19.293833 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet 400a0317f9184c99a449f7f160fa998f from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:37127: session id = f7f6592d13d245508261964ca6ffc5c4-400a0317f9184c99a449f7f160fa998f
I20240503 08:13:19.296171 32292 tablet_copy_source_session.cc:205] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.304695 2590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400a0317f9184c99a449f7f160fa998f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.307858 2597 ts_tablet_manager.cc:902] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:19.344772 2604 ts_tablet_manager.cc:902] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:19.346024 2590 tablet_copy_client.cc:783] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.346549 2590 tablet_copy_client.cc:647] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.349129 2594 tablet_copy_client.cc:1002] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:19.378304 2604 tablet_copy_client.cc:308] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:19.380195 2597 tablet_copy_client.cc:308] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:19.380190 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet 10646e4785b047d8a3ac6af5ddb06474 from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:46063)
I20240503 08:13:19.380375 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet 10646e4785b047d8a3ac6af5ddb06474 from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:46063: session id = e615bd1588964e2fbe14f14f4aae0a98-10646e4785b047d8a3ac6af5ddb06474
I20240503 08:13:19.383328 32161 tablet_copy_source_session.cc:205] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.383421 2619 ts_tablet_manager.cc:902] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:19.383992 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet b7996d398e3440ebbb8f942bda8f6833 from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:46063)
I20240503 08:13:19.384146 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet b7996d398e3440ebbb8f942bda8f6833 from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:46063: session id = e615bd1588964e2fbe14f14f4aae0a98-b7996d398e3440ebbb8f942bda8f6833
I20240503 08:13:19.384644 2604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10646e4785b047d8a3ac6af5ddb06474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.386819 32161 tablet_copy_source_session.cc:205] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.388015 2597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7996d398e3440ebbb8f942bda8f6833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.391546 2604 tablet_copy_client.cc:783] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.391842 2604 tablet_copy_client.cc:647] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.395236 2613 tablet_copy_client.cc:1002] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:19.395488 2597 tablet_copy_client.cc:783] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.395709 2597 tablet_copy_client.cc:647] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.397265 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:19.397795 2610 tablet_copy_client.cc:1002] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:19.398034 32251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:19.440310 32380 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:19.451309 2630 ts_tablet_manager.cc:902] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:19.456681 1705 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:19.460712 2619 tablet_copy_client.cc:308] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:19.461545 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet d9151dcb76174b27991fdcf3839db0a5 from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:49993)
I20240503 08:13:19.461709 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet d9151dcb76174b27991fdcf3839db0a5 from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:49993: session id = c6b280e8ecd54855b7b21b58a223149a-d9151dcb76174b27991fdcf3839db0a5
I20240503 08:13:19.465009 32292 tablet_copy_source_session.cc:205] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.466502 2619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9151dcb76174b27991fdcf3839db0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.478695 32119 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:19.502635 2619 tablet_copy_client.cc:783] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.503005 2619 tablet_copy_client.cc:647] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.504323 2630 tablet_copy_client.cc:308] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:19.505062 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet 8e6a81a3850e49edb08c8060a45b6831 from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:41801)
I20240503 08:13:19.505244 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet 8e6a81a3850e49edb08c8060a45b6831 from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:41801: session id = 6c93e47973e047d0a162a50f72f9a757-8e6a81a3850e49edb08c8060a45b6831
I20240503 08:13:19.508167 32292 tablet_copy_source_session.cc:205] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:19.510231 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a81a3850e49edb08c8060a45b6831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:19.504869 1571 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:19.516305 2632 tablet_copy_client.cc:1002] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:19.522600 2630 tablet_copy_client.cc:783] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:19.528811 2630 tablet_copy_client.cc:647] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:19.530844 1969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:19.535322 2637 tablet_copy_client.cc:1002] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:19.550132 1837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240503 08:13:19.743419 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.9:0
--local_ip_for_outbound_sockets=127.24.123.9
--webserver_interface=127.24.123.9
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44195
--builtin_ntp_servers=127.24.123.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 08:13:19.935564 2350 tablet_copy_client.cc:523] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:19.941504 2350 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:19.957137 2350 log.cc:824] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:20.010059 2361 tablet_copy_client.cc:523] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.013362 2361 tablet_bootstrap.cc:492] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:20.015556 2350 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.016088 2350 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:20.016546 2350 ts_tablet_manager.cc:1367] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.075s user 0.059s sys 0.000s
I20240503 08:13:20.023742 2350 raft_consensus.cc:348] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:20.024293 2350 raft_consensus.cc:729] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:20.024888 2350 consensus_queue.cc:260] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:20.032461 2350 ts_tablet_manager.cc:1398] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20240503 08:13:20.033669 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-2f4ac5dea8024445ad2cd1ca724f2290 received from {username='slave'} at 127.24.123.5:35625
I20240503 08:13:20.032368 2361 log.cc:824] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:20.033841 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-2f4ac5dea8024445ad2cd1ca724f2290 on tablet 2f4ac5dea8024445ad2cd1ca724f2290 with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:20.081691 2361 tablet_bootstrap.cc:492] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.082289 2361 tablet_bootstrap.cc:492] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:20.083629 2361 ts_tablet_manager.cc:1367] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.070s user 0.060s sys 0.001s
I20240503 08:13:20.090155 2361 raft_consensus.cc:348] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:20.090632 2361 raft_consensus.cc:729] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:20.091277 2361 consensus_queue.cc:260] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:20.099558 2361 ts_tablet_manager.cc:1398] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20240503 08:13:20.101831 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-27a469ffc17447ed86860e991ff3534a received from {username='slave'} at 127.24.123.8:35733
I20240503 08:13:20.102008 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-27a469ffc17447ed86860e991ff3534a on tablet 27a469ffc17447ed86860e991ff3534a with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:20.150074 2384 tablet_copy_client.cc:523] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:13:20.151202 2662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:20.151476 2662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:20.151577 2662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:13:20.154624 2384 tablet_bootstrap.cc:492] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
W20240503 08:13:20.161007 2662 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:20.161165 2662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:20.161381 2662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.9
I20240503 08:13:20.167896 2384 log.cc:824] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:20.174400 2662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.9:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.24.123.9
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.24.123.62:44195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.9
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:20.175928 2662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:20.177872 2662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:20.194051 2748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:20.194864 2749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:20.202025 2403 tablet_copy_client.cc:523] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.206435 2384 tablet_bootstrap.cc:492] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.206945 2384 tablet_bootstrap.cc:492] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:20.206956 2403 tablet_bootstrap.cc:492] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:20.207329 2384 ts_tablet_manager.cc:1367] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.053s user 0.037s sys 0.011s
I20240503 08:13:20.212241 2384 raft_consensus.cc:348] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:20.212613 2384 raft_consensus.cc:729] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:20.213093 2384 consensus_queue.cc:260] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:20.214776 2384 ts_tablet_manager.cc:1398] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20240503 08:13:20.216676 32422 tablet_copy_service.cc:342] P 71ad6bf07682497aaa1e9429320aef79: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-f6b5cbb6708049e6ae32da12bb27368f received from {username='slave'} at 127.24.123.6:47045
I20240503 08:13:20.216847 32422 tablet_copy_service.cc:433] P 71ad6bf07682497aaa1e9429320aef79: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-f6b5cbb6708049e6ae32da12bb27368f on tablet f6b5cbb6708049e6ae32da12bb27368f with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:20.218695 1725 raft_consensus.cc:1203] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.226058 2662 server_base.cc:1034] running on GCE node
I20240503 08:13:20.227586 2403 log.cc:824] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: Log is configured to *not* fsync() on all Append() calls
W20240503 08:13:20.228276 2751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:20.229355 2662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:20.239218 2662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:20.240536 2662 hybrid_clock.cc:648] HybridClock initialized: now 1714724000240419 us; error 98 us; skew 500 ppm
I20240503 08:13:20.241012 2662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:20.241441 2417 tablet_copy_client.cc:523] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.245471 2417 tablet_bootstrap.cc:492] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:20.267331 2662 webserver.cc:460] Webserver started at http://127.24.123.9:37743/ using document root <none> and password file <none>
I20240503 08:13:20.268538 2662 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:20.269099 2662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:20.269562 2662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:20.273133 2662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "237c0b0fe44549a89d1b5bfc8c55da91"
format_stamp: "Formatted at 2024-05-03 08:13:20 on dist-test-slave-gkrw"
I20240503 08:13:20.274117 2662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "237c0b0fe44549a89d1b5bfc8c55da91"
format_stamp: "Formatted at 2024-05-03 08:13:20 on dist-test-slave-gkrw"
I20240503 08:13:20.284353 2403 tablet_bootstrap.cc:492] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.285056 2403 tablet_bootstrap.cc:492] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:20.285635 2403 ts_tablet_manager.cc:1367] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.079s user 0.060s sys 0.004s
I20240503 08:13:20.292582 2403 raft_consensus.cc:348] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:20.293550 2403 raft_consensus.cc:729] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:20.294030 2403 consensus_queue.cc:260] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:20.296258 2662 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.011s sys 0.010s
I20240503 08:13:20.297009 2417 tablet_bootstrap.cc:492] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.297475 2417 tablet_bootstrap.cc:492] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:20.297768 2417 ts_tablet_manager.cc:1367] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.052s user 0.050s sys 0.001s
I20240503 08:13:20.298717 2417 raft_consensus.cc:348] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:20.299000 2417 raft_consensus.cc:729] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:20.299252 2417 consensus_queue.cc:260] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:20.299801 2417 ts_tablet_manager.cc:1398] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:20.300675 2770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:20.301110 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-2cf32f9e1e914653a6594d58c67cd80e received from {username='slave'} at 127.24.123.8:48191
I20240503 08:13:20.301292 2403 ts_tablet_manager.cc:1398] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20240503 08:13:20.301318 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-2cf32f9e1e914653a6594d58c67cd80e on tablet 2cf32f9e1e914653a6594d58c67cd80e with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:20.302506 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-afdc26bd51bd467f88d7892edb81ca9a received from {username='slave'} at 127.24.123.7:37127
I20240503 08:13:20.302645 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-afdc26bd51bd467f88d7892edb81ca9a on tablet afdc26bd51bd467f88d7892edb81ca9a with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:20.303229 2662 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20240503 08:13:20.303828 2662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "237c0b0fe44549a89d1b5bfc8c55da91"
format_stamp: "Formatted at 2024-05-03 08:13:20 on dist-test-slave-gkrw"
I20240503 08:13:20.304158 2662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:20.345036 2439 tablet_copy_client.cc:523] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.347995 2439 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
I20240503 08:13:20.351800 2662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:20.352952 2662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:20.353351 2662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:20.354703 2662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:20.360397 1989 raft_consensus.cc:1203] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.367713 2662 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:20.367837 2662 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:20.367957 2662 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:20.368031 2662 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:20.389317 2439 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.389796 2439 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:20.390058 2439 ts_tablet_manager.cc:1367] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.042s user 0.029s sys 0.007s
I20240503 08:13:20.390866 2439 raft_consensus.cc:348] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:20.391216 2439 raft_consensus.cc:729] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:20.391412 2439 consensus_queue.cc:260] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:20.391876 2439 ts_tablet_manager.cc:1398] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:20.392833 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-f9541e40695844a1b05ca1bc0dd8a10d received from {username='slave'} at 127.24.123.6:41801
I20240503 08:13:20.392983 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-f9541e40695844a1b05ca1bc0dd8a10d on tablet f9541e40695844a1b05ca1bc0dd8a10d with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:20.417678 2662 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.9:41169
I20240503 08:13:20.417728 2889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.9:41169 every 8 connection(s)
I20240503 08:13:20.419121 2662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 08:13:20.419464 2453 tablet_copy_client.cc:523] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.422756 2453 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:20.429029 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 2662
I20240503 08:13:20.429232 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 08:13:20.437548 2457 tablet_copy_client.cc:523] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.438267 2890 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44195
I20240503 08:13:20.438576 2890 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:20.439625 2890 heartbeater.cc:511] Master 127.24.123.62:44195 requested a full tablet report, sending...
I20240503 08:13:20.440925 2457 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
I20240503 08:13:20.441128 31992 ts_manager.cc:194] Registered new tserver with Master: 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169)
I20240503 08:13:20.442226 31992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.9:37647
I20240503 08:13:20.463070 2453 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.463608 2453 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:20.463943 2453 ts_tablet_manager.cc:1367] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.041s user 0.041s sys 0.001s
I20240503 08:13:20.464885 2453 raft_consensus.cc:348] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:20.465104 2453 raft_consensus.cc:729] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:20.465327 2453 consensus_queue.cc:260] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:20.465860 2453 ts_tablet_manager.cc:1398] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:20.466836 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-2bd736780ea840ab9b6630a4acd80ae9 received from {username='slave'} at 127.24.123.5:56869
I20240503 08:13:20.467033 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-2bd736780ea840ab9b6630a4acd80ae9 on tablet 2bd736780ea840ab9b6630a4acd80ae9 with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:20.477440 2457 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.477864 2457 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:20.478120 2457 ts_tablet_manager.cc:1367] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.037s user 0.033s sys 0.001s
I20240503 08:13:20.478717 2457 raft_consensus.cc:348] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:20.479068 2457 raft_consensus.cc:729] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:20.479269 2457 consensus_queue.cc:260] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:20.480393 2457 ts_tablet_manager.cc:1398] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 08:13:20.481001 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-734d2de8d2804b2c8d2f144157e12f77 received from {username='slave'} at 127.24.123.6:52495
I20240503 08:13:20.481137 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-734d2de8d2804b2c8d2f144157e12f77 on tablet 734d2de8d2804b2c8d2f144157e12f77 with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:20.486768 1989 raft_consensus.cc:1203] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.558431 1857 raft_consensus.cc:1203] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.562731 2474 tablet_copy_client.cc:523] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.565402 2474 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:20.569586 1591 raft_consensus.cc:1203] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.582820 1591 raft_consensus.cc:1203] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.596876 2493 tablet_copy_client.cc:523] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.600661 2493 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:20.605446 2474 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.605937 2474 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:20.606192 2474 ts_tablet_manager.cc:1367] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.041s user 0.033s sys 0.007s
I20240503 08:13:20.607156 2474 raft_consensus.cc:348] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:20.607455 2474 raft_consensus.cc:729] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:20.607683 2474 consensus_queue.cc:260] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:20.608865 2474 ts_tablet_manager.cc:1398] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:20.609880 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-25f1956062e84b92a3b036fc5dff4b46 received from {username='slave'} at 127.24.123.7:37127
I20240503 08:13:20.610014 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-25f1956062e84b92a3b036fc5dff4b46 on tablet 25f1956062e84b92a3b036fc5dff4b46 with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:20.613612 2121 raft_consensus.cc:1050] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:20.614696 2121 consensus_queue.cc:237] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:20.617921 1725 raft_consensus.cc:1261] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.618052 32532 raft_consensus.cc:1261] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.618619 2388 consensus_queue.cc:1035] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.618968 2391 consensus_queue.cc:1035] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.624904 32138 raft_consensus.cc:1261] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.625517 2388 consensus_queue.cc:1035] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.639582 32530 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:20.647372 32000 catalog_manager.cc:5484] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:20.649363 2511 tablet_copy_client.cc:523] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.638475 2121 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:20.652911 2511 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:20.655943 2493 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.656424 2493 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:20.656561 2485 tablet_copy_client.cc:523] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.656720 2493 ts_tablet_manager.cc:1367] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.056s user 0.050s sys 0.005s
I20240503 08:13:20.657550 2493 raft_consensus.cc:348] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:20.656881 1725 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:20.657795 2493 raft_consensus.cc:729] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:20.657980 2493 consensus_queue.cc:260] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:20.658092 32138 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:20.659906 2493 ts_tablet_manager.cc:1398] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:20.660569 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-7cc8a3241af248578b6ce87b1e33cea9 received from {username='slave'} at 127.24.123.8:46063
I20240503 08:13:20.660683 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-7cc8a3241af248578b6ce87b1e33cea9 on tablet 7cc8a3241af248578b6ce87b1e33cea9 with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:20.661376 2485 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
I20240503 08:13:20.681651 1725 raft_consensus.cc:1203] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.692044 32400 consensus_queue.cc:237] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:20.695806 32532 raft_consensus.cc:1261] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:20.696854 2388 consensus_queue.cc:1035] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:20.698556 1725 raft_consensus.cc:1261] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:20.700294 2388 consensus_queue.cc:1035] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:20.702107 2388 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:20.707065 32532 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:20.711679 1725 raft_consensus.cc:2932] T f6b5cbb6708049e6ae32da12bb27368f P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:20.715678 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6b5cbb6708049e6ae32da12bb27368f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:20.717943 31992 catalog_manager.cc:5484] T f6b5cbb6708049e6ae32da12bb27368f P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:20.726730 2511 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.727239 2511 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:20.727574 2511 ts_tablet_manager.cc:1367] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.075s user 0.046s sys 0.005s
I20240503 08:13:20.727600 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b5cbb6708049e6ae32da12bb27368f with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:20.728394 2511 raft_consensus.cc:348] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:20.728767 2511 raft_consensus.cc:729] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:20.729038 2511 consensus_queue.cc:260] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:20.731185 2511 ts_tablet_manager.cc:1398] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:13:20.731449 2953 tablet_replica.cc:330] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:20.731901 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-5de086cc1f0b497f87b055e745a38e0d received from {username='slave'} at 127.24.123.5:49993
I20240503 08:13:20.732055 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-5de086cc1f0b497f87b055e745a38e0d on tablet 5de086cc1f0b497f87b055e745a38e0d with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:20.731951 2953 raft_consensus.cc:2229] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:20.732579 2953 raft_consensus.cc:2258] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:20.737448 2953 ts_tablet_manager.cc:1875] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:20.737582 1725 raft_consensus.cc:1203] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.744414 2953 ts_tablet_manager.cc:1888] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:20.744633 2953 log.cc:1196] T f6b5cbb6708049e6ae32da12bb27368f P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/f6b5cbb6708049e6ae32da12bb27368f
I20240503 08:13:20.745571 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet f6b5cbb6708049e6ae32da12bb27368f (table rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]) successfully deleted
I20240503 08:13:20.745890 2485 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.746388 2485 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:20.746680 2485 ts_tablet_manager.cc:1367] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.085s user 0.039s sys 0.012s
I20240503 08:13:20.747355 2485 raft_consensus.cc:348] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:20.747577 2485 raft_consensus.cc:729] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:20.747786 2485 consensus_queue.cc:260] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:20.748337 2485 ts_tablet_manager.cc:1398] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:20.748538 2539 tablet_copy_client.cc:523] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.751312 2539 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:20.755415 32422 tablet_copy_service.cc:342] P 71ad6bf07682497aaa1e9429320aef79: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-95491576997c4392b482b4627760d818 received from {username='slave'} at 127.24.123.6:47045
I20240503 08:13:20.755564 32422 tablet_copy_service.cc:433] P 71ad6bf07682497aaa1e9429320aef79: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-95491576997c4392b482b4627760d818 on tablet 95491576997c4392b482b4627760d818 with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:20.786851 2539 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.787436 2539 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:20.787684 2539 ts_tablet_manager.cc:1367] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.036s user 0.030s sys 0.004s
I20240503 08:13:20.788336 2539 raft_consensus.cc:348] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:20.788612 2539 raft_consensus.cc:729] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:20.788820 2539 consensus_queue.cc:260] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:20.789378 2539 ts_tablet_manager.cc:1398] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:20.790364 32422 tablet_copy_service.cc:342] P 71ad6bf07682497aaa1e9429320aef79: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-af5c3865ca694bf090dce19f1100e5eb received from {username='slave'} at 127.24.123.7:48777
I20240503 08:13:20.790493 32422 tablet_copy_service.cc:433] P 71ad6bf07682497aaa1e9429320aef79: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-af5c3865ca694bf090dce19f1100e5eb on tablet af5c3865ca694bf090dce19f1100e5eb with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:20.807848 1857 raft_consensus.cc:1203] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.849484 2590 tablet_copy_client.cc:523] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.852707 2590 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:20.857416 2041 raft_consensus.cc:1050] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:20.858572 2041 consensus_queue.cc:237] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:20.863978 1989 raft_consensus.cc:1261] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.864110 32400 raft_consensus.cc:1261] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.864223 32271 raft_consensus.cc:1261] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.864786 3002 consensus_queue.cc:1035] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.865150 3005 consensus_queue.cc:1035] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.866240 2927 consensus_queue.cc:1035] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.879407 1989 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.879921 32272 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.883653 1725 raft_consensus.cc:1203] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:20.878523 3002 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.885011 31994 catalog_manager.cc:5484] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.891485 32400 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.895732 2604 tablet_copy_client.cc:523] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.896440 1906 raft_consensus.cc:1050] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:20.897413 1906 consensus_queue.cc:237] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:20.898597 2597 tablet_copy_client.cc:523] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:20.901832 2597 tablet_bootstrap.cc:492] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:20.905002 2604 tablet_bootstrap.cc:492] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:20.908475 2590 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.908891 2590 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:20.909179 2590 ts_tablet_manager.cc:1367] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.057s user 0.047s sys 0.000s
I20240503 08:13:20.910048 2590 raft_consensus.cc:348] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:20.910331 2590 raft_consensus.cc:729] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:20.910550 2590 consensus_queue.cc:260] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:20.911947 32400 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.912460 1988 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.912456 32138 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.913794 2042 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.913913 2590 ts_tablet_manager.cc:1398] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:13:20.914426 2042 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.914933 2042 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.917155 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-400a0317f9184c99a449f7f160fa998f received from {username='slave'} at 127.24.123.7:37127
I20240503 08:13:20.917291 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-400a0317f9184c99a449f7f160fa998f on tablet 400a0317f9184c99a449f7f160fa998f with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:20.938076 32532 consensus_queue.cc:237] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:20.938783 3009 raft_consensus.cc:1050] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:20.939991 32401 raft_consensus.cc:1261] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:20.940146 3009 consensus_queue.cc:237] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:20.940618 2927 consensus_queue.cc:1035] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:20.942118 1988 raft_consensus.cc:1261] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:20.943317 2927 consensus_queue.cc:1035] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:20.944427 3002 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.945089 32401 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.948953 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2cf32f9e1e914653a6594d58c67cd80e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:20.948916 1988 raft_consensus.cc:2932] T 2cf32f9e1e914653a6594d58c67cd80e P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.950554 31994 catalog_manager.cc:5484] T 2cf32f9e1e914653a6594d58c67cd80e P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:20.955826 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf32f9e1e914653a6594d58c67cd80e with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:20.963196 1906 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.965307 1857 raft_consensus.cc:1261] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.965967 3009 consensus_queue.cc:1035] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.966208 32138 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.971997 31994 catalog_manager.cc:5484] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.972013 32401 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:20.978926 3020 tablet_replica.cc:330] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:20.979512 3020 raft_consensus.cc:2229] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:20.979624 32138 raft_consensus.cc:1261] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.980109 3020 raft_consensus.cc:2258] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:20.981158 3009 consensus_queue.cc:1035] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.981792 2597 tablet_bootstrap.cc:492] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:20.981806 3020 ts_tablet_manager.cc:1875] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:20.982198 2597 tablet_bootstrap.cc:492] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:20.982810 2597 ts_tablet_manager.cc:1367] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.081s user 0.046s sys 0.011s
I20240503 08:13:20.983758 2597 raft_consensus.cc:348] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:20.983996 2597 raft_consensus.cc:729] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:20.984210 2597 consensus_queue.cc:260] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:20.985348 2597 ts_tablet_manager.cc:1398] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:20.986044 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-b7996d398e3440ebbb8f942bda8f6833 received from {username='slave'} at 127.24.123.8:46063
I20240503 08:13:20.986151 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-b7996d398e3440ebbb8f942bda8f6833 on tablet b7996d398e3440ebbb8f942bda8f6833 with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:20.987129 32532 raft_consensus.cc:1261] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:20.988059 3009 consensus_queue.cc:1035] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:20.990520 3020 ts_tablet_manager.cc:1888] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:20.990798 3020 log.cc:1196] T 2cf32f9e1e914653a6594d58c67cd80e P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/2cf32f9e1e914653a6594d58c67cd80e
I20240503 08:13:20.990691 3012 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:20.991425 32138 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:20.991889 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet 2cf32f9e1e914653a6594d58c67cd80e (table rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]) successfully deleted
I20240503 08:13:20.994071 32532 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:20.995793 1857 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:20.997220 31994 catalog_manager.cc:5484] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:21.002709 1988 raft_consensus.cc:1203] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:21.003557 32272 consensus_queue.cc:237] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:21.004565 1989 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.014243 1856 raft_consensus.cc:1261] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.014320 32138 raft_consensus.cc:1261] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.016013 3012 consensus_queue.cc:1035] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.016209 1591 raft_consensus.cc:1203] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:21.016633 2619 tablet_copy_client.cc:523] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:21.017024 3024 consensus_queue.cc:1035] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.021003 2619 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:21.020742 3012 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.024636 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet afdc26bd51bd467f88d7892edb81ca9a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.024695 1856 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.026996 1857 raft_consensus.cc:1203] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:21.030977 31992 catalog_manager.cc:5484] T afdc26bd51bd467f88d7892edb81ca9a P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:21.035535 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet afdc26bd51bd467f88d7892edb81ca9a with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:21.036209 2630 tablet_copy_client.cc:523] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:21.037804 32138 raft_consensus.cc:2932] T afdc26bd51bd467f88d7892edb81ca9a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.038738 2630 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
I20240503 08:13:21.047083 2604 tablet_bootstrap.cc:492] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:21.047518 2604 tablet_bootstrap.cc:492] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:21.047793 2604 ts_tablet_manager.cc:1367] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.143s user 0.052s sys 0.022s
I20240503 08:13:21.048593 2604 raft_consensus.cc:348] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:21.048861 2604 raft_consensus.cc:729] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:21.049043 2604 consensus_queue.cc:260] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:21.049572 2604 ts_tablet_manager.cc:1398] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:21.050938 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-10646e4785b047d8a3ac6af5ddb06474 received from {username='slave'} at 127.24.123.8:46063
I20240503 08:13:21.051069 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-10646e4785b047d8a3ac6af5ddb06474 on tablet 10646e4785b047d8a3ac6af5ddb06474 with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:21.057068 3026 tablet_replica.cc:330] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:21.057904 3026 raft_consensus.cc:2229] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.059357 3026 raft_consensus.cc:2258] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.067745 3026 ts_tablet_manager.cc:1875] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.077035 3026 ts_tablet_manager.cc:1888] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.077263 3026 log.cc:1196] T afdc26bd51bd467f88d7892edb81ca9a P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/afdc26bd51bd467f88d7892edb81ca9a
I20240503 08:13:21.078233 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet afdc26bd51bd467f88d7892edb81ca9a (table rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]) successfully deleted
I20240503 08:13:21.109436 1856 raft_consensus.cc:1203] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:21.109735 2630 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:21.110234 2630 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:21.110515 2630 ts_tablet_manager.cc:1367] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.072s user 0.048s sys 0.006s
I20240503 08:13:21.111451 2630 raft_consensus.cc:348] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:21.111925 2630 raft_consensus.cc:729] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:21.112284 2630 consensus_queue.cc:260] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:21.112848 2630 ts_tablet_manager.cc:1398] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:21.113586 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-8e6a81a3850e49edb08c8060a45b6831 received from {username='slave'} at 127.24.123.6:41801
I20240503 08:13:21.113709 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-8e6a81a3850e49edb08c8060a45b6831 on tablet 8e6a81a3850e49edb08c8060a45b6831 with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:21.117281 2619 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:21.117708 2619 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:21.117991 2619 ts_tablet_manager.cc:1367] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.097s user 0.047s sys 0.007s
I20240503 08:13:21.118767 2619 raft_consensus.cc:348] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:21.119030 2619 raft_consensus.cc:729] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:21.119323 2619 consensus_queue.cc:260] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:21.119976 2619 ts_tablet_manager.cc:1398] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:21.120738 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-d9151dcb76174b27991fdcf3839db0a5 received from {username='slave'} at 127.24.123.5:49993
I20240503 08:13:21.120862 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-d9151dcb76174b27991fdcf3839db0a5 on tablet d9151dcb76174b27991fdcf3839db0a5 with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:21.128012 3024 raft_consensus.cc:1050] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:21.128916 3024 consensus_queue.cc:237] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:21.131548 32138 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.131736 32401 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.131953 1725 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.132406 1906 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.132700 3009 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.132997 3024 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.137387 1906 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.140343 32401 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.145947 31992 catalog_manager.cc:5484] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.148176 32138 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.149914 1725 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.152529 32272 consensus_queue.cc:237] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:21.153611 2927 raft_consensus.cc:1050] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:21.154457 2927 consensus_queue.cc:237] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:21.155642 32138 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.156198 2042 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.156466 1724 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.157013 2042 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.159268 3012 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.161002 32138 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.163100 32141 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.163317 1725 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.164172 32401 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.164321 1591 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.165361 2041 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.165681 3010 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.166162 3002 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.167876 31992 catalog_manager.cc:5484] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.172395 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet f9541e40695844a1b05ca1bc0dd8a10d with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:21.173269 3057 tablet_replica.cc:330] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:21.173761 3057 raft_consensus.cc:2229] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.174034 3002 raft_consensus.cc:1050] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:21.174219 3057 raft_consensus.cc:2258] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.174904 3002 consensus_queue.cc:237] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:21.177631 3057 ts_tablet_manager.cc:1875] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.177448 2927 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.181803 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9541e40695844a1b05ca1bc0dd8a10d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.181684 1591 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.183017 3057 ts_tablet_manager.cc:1888] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.183203 3057 log.cc:1196] T f9541e40695844a1b05ca1bc0dd8a10d P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/f9541e40695844a1b05ca1bc0dd8a10d
I20240503 08:13:21.186014 2331 raft_consensus.cc:1050] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:21.186942 2331 consensus_queue.cc:237] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:21.187919 32401 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.188653 32138 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.191653 32272 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.191681 1725 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.192227 2041 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.192797 2927 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.193593 32138 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.194082 2041 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.197154 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet f9541e40695844a1b05ca1bc0dd8a10d (table rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]) successfully deleted
I20240503 08:13:21.199968 32000 catalog_manager.cc:5484] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.205663 32530 consensus_queue.cc:237] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:21.207991 32138 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.208518 3005 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.211905 32530 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.212069 3005 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.212677 2328 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.215601 32271 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.215977 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f4ac5dea8024445ad2cd1ca724f2290 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.217088 32140 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.220672 32270 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.222179 1589 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.222661 32000 catalog_manager.cc:5484] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.222997 2328 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.223532 2328 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.227120 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4ac5dea8024445ad2cd1ca724f2290 with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:21.227531 3057 tablet_replica.cc:330] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:21.227778 3057 raft_consensus.cc:2229] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.228055 3057 raft_consensus.cc:2258] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.227823 32140 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.214633 2041 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.234516 3057 ts_tablet_manager.cc:1875] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.235448 1590 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.238112 32000 catalog_manager.cc:5484] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.241174 3057 ts_tablet_manager.cc:1888] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.241303 3057 log.cc:1196] T 2f4ac5dea8024445ad2cd1ca724f2290 P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/2f4ac5dea8024445ad2cd1ca724f2290
I20240503 08:13:21.241426 1989 raft_consensus.cc:1203] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:21.241948 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet 2f4ac5dea8024445ad2cd1ca724f2290 (table rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]) successfully deleted
I20240503 08:13:21.244077 3010 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.246052 1725 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.249815 32530 consensus_queue.cc:237] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:21.254247 32141 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.254812 3005 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.257282 1725 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.257886 3005 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.257737 2041 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.259537 1725 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.259862 1591 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.260375 2328 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.260502 1590 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.265799 31996 catalog_manager.cc:5484] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.265841 32272 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.268378 2042 raft_consensus.cc:1050] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:21.269165 2042 consensus_queue.cc:237] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:21.271360 31992 catalog_manager.cc:5484] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.275313 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet 734d2de8d2804b2c8d2f144157e12f77 with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:21.275585 1856 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.275648 3020 tablet_replica.cc:330] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:21.275856 3020 raft_consensus.cc:2229] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.276130 3020 raft_consensus.cc:2258] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.276258 32141 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.276427 32530 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.276876 1906 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.277374 1906 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.277688 3020 ts_tablet_manager.cc:1875] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.278662 3012 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.283443 32140 consensus_queue.cc:237] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:21.284735 32530 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.289518 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 734d2de8d2804b2c8d2f144157e12f77 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.303120 3024 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.306354 1856 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.310230 31996 catalog_manager.cc:5484] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:21.312103 3020 ts_tablet_manager.cc:1888] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.311844 32530 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.312405 3020 log.cc:1196] T 734d2de8d2804b2c8d2f144157e12f77 P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/734d2de8d2804b2c8d2f144157e12f77
I20240503 08:13:21.315907 32272 consensus_queue.cc:237] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:21.316998 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet 734d2de8d2804b2c8d2f144157e12f77 (table rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]) successfully deleted
I20240503 08:13:21.319168 2388 raft_consensus.cc:1050] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:21.319890 1856 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.320096 2388 consensus_queue.cc:237] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:21.320711 3024 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.323935 32532 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.324775 3024 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.325299 3024 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.327507 32532 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.328859 1856 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.329321 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25f1956062e84b92a3b036fc5dff4b46 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.331435 31992 catalog_manager.cc:5484] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:21.334415 1725 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.335086 2391 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.337934 32532 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.338716 2388 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.338795 32141 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
W20240503 08:13:21.339392 32074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:34237, user_credentials={real_user=slave}} blocked reactor thread for 52427.5us
I20240503 08:13:21.340669 32138 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.341105 32139 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.343240 1591 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.344800 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f1956062e84b92a3b036fc5dff4b46 with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:21.345000 32270 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.346601 3081 tablet_replica.cc:330] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:21.346654 2331 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.346952 3081 raft_consensus.cc:2229] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.347311 3081 raft_consensus.cc:2258] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.348268 3081 ts_tablet_manager.cc:1875] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.348886 1591 raft_consensus.cc:1203] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:21.348901 2331 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.350432 32272 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.355396 31994 catalog_manager.cc:5484] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.356884 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2bd736780ea840ab9b6630a4acd80ae9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.357195 3081 ts_tablet_manager.cc:1888] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.357362 3081 log.cc:1196] T 25f1956062e84b92a3b036fc5dff4b46 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/25f1956062e84b92a3b036fc5dff4b46
I20240503 08:13:21.358372 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet 25f1956062e84b92a3b036fc5dff4b46 (table rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]) successfully deleted
I20240503 08:13:21.359439 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bd736780ea840ab9b6630a4acd80ae9 with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:21.359799 3026 tablet_replica.cc:330] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:21.359982 3026 raft_consensus.cc:2229] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.360209 3026 raft_consensus.cc:2258] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.360657 2391 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.361320 3026 ts_tablet_manager.cc:1875] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.361704 2388 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.373684 3026 ts_tablet_manager.cc:1888] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.374255 3077 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.374805 3026 log.cc:1196] T 2bd736780ea840ab9b6630a4acd80ae9 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/2bd736780ea840ab9b6630a4acd80ae9
I20240503 08:13:21.375633 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet 2bd736780ea840ab9b6630a4acd80ae9 (table rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]) successfully deleted
I20240503 08:13:21.375751 1591 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.378302 32532 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.379191 1725 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.379288 31994 catalog_manager.cc:5484] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:21.382120 32141 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.391069 32401 consensus_queue.cc:237] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:21.392941 1725 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.393435 3083 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.393498 32532 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.394130 3083 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.395350 3084 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.398402 1725 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.397541 32532 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:21.398962 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95491576997c4392b482b4627760d818 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.403030 31992 catalog_manager.cc:5484] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 5 to 6, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:21.407915 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet 95491576997c4392b482b4627760d818 with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:21.408208 3081 tablet_replica.cc:330] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:21.408385 3081 raft_consensus.cc:2229] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.408713 3081 raft_consensus.cc:2258] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.409461 3081 ts_tablet_manager.cc:1875] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.413445 3081 ts_tablet_manager.cc:1888] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.413578 3081 log.cc:1196] T 95491576997c4392b482b4627760d818 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/95491576997c4392b482b4627760d818
I20240503 08:13:21.414223 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet 95491576997c4392b482b4627760d818 (table rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]) successfully deleted
I20240503 08:13:21.445488 2890 heartbeater.cc:503] Master 127.24.123.62:44195 was elected leader, sending a full tablet report...
I20240503 08:13:21.458278 32272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27a469ffc17447ed86860e991ff3534a"
dest_uuid: "272c59640332489181cbf2cc52d266bb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42446
I20240503 08:13:21.458498 32272 raft_consensus.cc:595] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:21.473803 3012 raft_consensus.cc:1050] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:21.474610 3012 consensus_queue.cc:237] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:21.477546 32141 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.477807 32532 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.478159 1906 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.478448 1591 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.478669 1906 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.479624 2042 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.482527 1906 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.486166 32139 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.486614 32532 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.487993 31992 catalog_manager.cc:5484] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:21.495083 1989 raft_consensus.cc:1203] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:21.501817 1591 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.502264 32272 consensus_queue.cc:237] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:21.505224 32141 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.505311 1589 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.506449 2042 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.506773 3024 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.506893 2042 raft_consensus.cc:979] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: : Instructing follower 99e2f0f92de9442c9e73d00d03ca6b90 to start an election
I20240503 08:13:21.507017 2042 raft_consensus.cc:1067] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Signalling peer 99e2f0f92de9442c9e73d00d03ca6b90 to start an election
I20240503 08:13:21.507959 32139 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27a469ffc17447ed86860e991ff3534a"
dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
from {username='slave'} at 127.24.123.2:60413
I20240503 08:13:21.508174 32139 raft_consensus.cc:482] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:21.508316 32139 raft_consensus.cc:3037] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:21.509930 3009 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.510895 1589 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.510860 32141 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.511394 32139 raft_consensus.cc:504] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:21.513238 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a469ffc17447ed86860e991ff3534a" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e615bd1588964e2fbe14f14f4aae0a98"
I20240503 08:13:21.513404 32401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a469ffc17447ed86860e991ff3534a" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:21.513569 1989 raft_consensus.cc:3037] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:21.513660 32401 raft_consensus.cc:3037] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:21.514039 32272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a469ffc17447ed86860e991ff3534a" candidate_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:21.514233 32272 raft_consensus.cc:3032] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:21.514308 32272 raft_consensus.cc:729] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:21.514611 32272 consensus_queue.cc:260] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:21.515093 32272 raft_consensus.cc:3037] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:21.515898 32401 raft_consensus.cc:2454] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 2.
I20240503 08:13:21.516377 1989 raft_consensus.cc:2454] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 2.
I20240503 08:13:21.516577 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5de086cc1f0b497f87b055e745a38e0d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.517168 32076 leader_election.cc:302] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90, e615bd1588964e2fbe14f14f4aae0a98; no voters:
I20240503 08:13:21.517458 3077 raft_consensus.cc:2781] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:21.517539 32272 raft_consensus.cc:2454] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e2f0f92de9442c9e73d00d03ca6b90 in term 2.
I20240503 08:13:21.517525 32139 leader_election.cc:288] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [CANDIDATE]: Term 2 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749)
I20240503 08:13:21.517635 3077 raft_consensus.cc:686] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 LEADER]: Becoming Leader. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:21.517948 3077 consensus_queue.cc:237] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:21.518832 32000 catalog_manager.cc:5484] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:21.522429 31994 catalog_manager.cc:5484] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: term changed from 1 to 2, leader changed from 272c59640332489181cbf2cc52d266bb (127.24.123.2) to 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1). New cstate: current_term: 2 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:21.524760 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de086cc1f0b497f87b055e745a38e0d with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:21.525084 3026 tablet_replica.cc:330] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:21.525265 3026 raft_consensus.cc:2229] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.525485 3026 raft_consensus.cc:2258] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.526588 3026 ts_tablet_manager.cc:1875] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.533304 3026 ts_tablet_manager.cc:1888] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.533461 3026 log.cc:1196] T 5de086cc1f0b497f87b055e745a38e0d P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/5de086cc1f0b497f87b055e745a38e0d
I20240503 08:13:21.534166 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet 5de086cc1f0b497f87b055e745a38e0d (table rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]) successfully deleted
I20240503 08:13:21.581300 1725 raft_consensus.cc:1203] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:21.585866 3070 raft_consensus.cc:1050] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:21.586795 3070 consensus_queue.cc:237] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:21.589617 1989 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.589957 32272 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.590165 32401 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.590557 3077 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.590925 3070 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.591315 2331 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.595890 3070 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.597244 32401 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.599826 32272 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.603056 31994 catalog_manager.cc:5484] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.603642 1989 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.612771 32141 consensus_queue.cc:237] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:21.614553 1989 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.614630 32272 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.615161 3076 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.615640 3070 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.617434 3076 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.618155 32272 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.621099 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7cc8a3241af248578b6ce87b1e33cea9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.621130 1989 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.623071 31996 catalog_manager.cc:5484] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:21.624109 3012 raft_consensus.cc:1050] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:21.624858 3012 consensus_queue.cc:237] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:21.628196 32141 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.628482 32401 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.628861 1906 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.629033 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cc8a3241af248578b6ce87b1e33cea9 with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:21.629344 3057 tablet_replica.cc:330] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:21.629351 3024 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.629539 3057 raft_consensus.cc:2229] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.629781 3057 raft_consensus.cc:2258] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.630059 1856 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.631538 3024 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.632340 3057 ts_tablet_manager.cc:1875] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.639951 1906 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.640611 32401 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.644127 1856 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.646579 31992 catalog_manager.cc:5484] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.658138 32139 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.659375 3057 ts_tablet_manager.cc:1888] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.659606 3057 log.cc:1196] T 7cc8a3241af248578b6ce87b1e33cea9 P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/7cc8a3241af248578b6ce87b1e33cea9
I20240503 08:13:21.660398 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet 7cc8a3241af248578b6ce87b1e33cea9 (table rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]) successfully deleted
I20240503 08:13:21.662930 32141 consensus_queue.cc:237] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:21.664876 32272 consensus_queue.cc:237] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:21.666391 32139 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.666551 1856 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.667119 3012 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.667678 3012 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.669564 3024 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.671700 32139 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.673907 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 400a0317f9184c99a449f7f160fa998f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.675035 3083 raft_consensus.cc:1050] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:21.675827 3083 consensus_queue.cc:237] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:21.678185 31996 catalog_manager.cc:5484] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.680723 32139 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.681771 32272 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.682166 3084 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.675868 1856 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.683902 1857 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.684434 2391 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.684783 32401 raft_consensus.cc:1261] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:21.684922 2391 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.685362 3076 consensus_queue.cc:1035] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.689134 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet 400a0317f9184c99a449f7f160fa998f with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:21.691648 32270 raft_consensus.cc:1261] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:21.692445 32074 consensus_peers.cc:485] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d3ee53044cd43debe6b59a2ef51c230. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:21.692704 3076 consensus_queue.cc:1035] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.697760 3057 tablet_replica.cc:330] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:21.698022 3057 raft_consensus.cc:2229] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.698320 3057 raft_consensus.cc:2258] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.699258 3057 ts_tablet_manager.cc:1875] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.702865 2331 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:21.703600 32401 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:21.708962 32000 catalog_manager.cc:5484] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:21.713119 32270 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:21.714956 3084 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.719516 1856 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.719472 32270 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.724203 31996 catalog_manager.cc:5484] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:21.740439 3057 ts_tablet_manager.cc:1888] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.741506 3057 log.cc:1196] T 400a0317f9184c99a449f7f160fa998f P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/400a0317f9184c99a449f7f160fa998f
I20240503 08:13:21.742601 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet 400a0317f9184c99a449f7f160fa998f (table rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]) successfully deleted
I20240503 08:13:21.743202 32139 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.745957 32401 consensus_queue.cc:237] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:21.748139 32141 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.748147 1856 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:21.748963 3084 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.749190 2391 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.748521 32139 consensus_queue.cc:237] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:21.751645 2391 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.752291 1856 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.755671 32141 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:21.755997 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af5c3865ca694bf090dce19f1100e5eb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:21.757519 3075 raft_consensus.cc:1050] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:21.758386 3075 consensus_queue.cc:237] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:21.760344 32532 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:21.760928 3077 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.762215 32000 catalog_manager.cc:5484] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:21.763979 32401 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:21.764570 2331 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.768218 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet af5c3865ca694bf090dce19f1100e5eb with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:21.768604 3020 tablet_replica.cc:330] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:21.768792 3020 raft_consensus.cc:2229] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:21.769061 3020 raft_consensus.cc:2258] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:21.769953 3020 ts_tablet_manager.cc:1875] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:21.777179 3020 ts_tablet_manager.cc:1888] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:21.778633 3020 log.cc:1196] T af5c3865ca694bf090dce19f1100e5eb P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/af5c3865ca694bf090dce19f1100e5eb
I20240503 08:13:21.779945 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet af5c3865ca694bf090dce19f1100e5eb (table rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]) successfully deleted
I20240503 08:13:21.784196 2331 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:21.789839 32400 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:21.794530 32532 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:21.795078 32000 catalog_manager.cc:5484] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:21.803004 32401 raft_consensus.cc:1261] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.803551 2331 consensus_queue.cc:1035] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.809118 32532 raft_consensus.cc:1261] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.809759 3075 consensus_queue.cc:1035] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.815924 3077 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.816591 1989 raft_consensus.cc:1261] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.817206 3076 consensus_queue.cc:1035] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.819073 1989 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.820811 32532 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.825191 32000 catalog_manager.cc:5484] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:21.828840 32076 consensus_peers.cc:485] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86b7e012f1544986a9460cd5dac5cece. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:21.828994 32400 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.847744 32532 consensus_queue.cc:237] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:21.850445 32400 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:21.850468 32466 consensus_peers.cc:485] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c970023069d44323a2b38629da1e2fad. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:21.851002 32139 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:21.851028 3010 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.852145 3010 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.855048 3005 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:21.855667 32139 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:21.855839 32400 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:21.861079 31996 catalog_manager.cc:5484] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:21.886999 32270 consensus_queue.cc:237] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:21.889010 32530 raft_consensus.cc:1261] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:21.889230 32207 consensus_peers.cc:485] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 130b43c1e784400fb8851a71428014a1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:21.889551 32400 raft_consensus.cc:1261] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:21.889533 3024 consensus_queue.cc:1035] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.890061 3024 consensus_queue.cc:1035] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.892042 2042 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:21.892608 32530 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:21.893188 32400 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:21.897653 31992 catalog_manager.cc:5484] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:21.898969 1989 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:21.899772 3075 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.916095 32530 consensus_queue.cc:237] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:21.918125 32400 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:21.918248 32270 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:21.918457 32466 consensus_peers.cc:485] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59a99f630f59472ca5c6778525e79fbe. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:21.918761 2041 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.919257 3010 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.921671 3010 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:21.923740 32270 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:21.923928 32400 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:21.928704 31996 catalog_manager.cc:5484] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:21.942814 32400 consensus_queue.cc:237] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:21.945127 32270 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:21.945333 32530 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:21.945744 2391 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.946172 3084 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:21.948556 2391 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
W20240503 08:13:21.953223 32337 consensus_peers.cc:485] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4e25ba167a248ce883fbafc4f6f5fd6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:21.955410 32270 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:21.956573 32000 catalog_manager.cc:5484] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:21.960258 32530 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:21.965176 32400 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:21.966074 3075 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.970618 1906 raft_consensus.cc:1050] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:21.971470 1906 consensus_queue.cc:237] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:21.973170 32270 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:21.974087 2331 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:21.974212 32139 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.974714 32530 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.975142 1589 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:21.976626 3012 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.978382 32139 consensus_queue.cc:237] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:21.980342 3012 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.981456 3012 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:21.981889 1989 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:13:21.982411 2331 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:21.983687 32401 raft_consensus.cc:1261] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:13:21.984797 2331 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.987800 3077 consensus_queue.cc:1035] T 27a469ffc17447ed86860e991ff3534a P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:21.989436 1989 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.989599 32401 raft_consensus.cc:2932] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:21.996685 1906 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.997685 32139 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:21.999130 31994 catalog_manager.cc:5484] T 27a469ffc17447ed86860e991ff3534a P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 2 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.001006 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27a469ffc17447ed86860e991ff3534a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:22.003813 1589 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:22.007841 32000 catalog_manager.cc:5484] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:22.013212 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a469ffc17447ed86860e991ff3534a with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:22.013960 3020 tablet_replica.cc:330] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:22.014181 3020 raft_consensus.cc:2229] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:22.014369 3020 raft_consensus.cc:2258] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:22.015753 3020 ts_tablet_manager.cc:1875] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:22.020398 32270 consensus_queue.cc:237] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:22.022481 1589 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:22.023545 3012 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:22.023623 32139 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:22.025035 3020 ts_tablet_manager.cc:1888] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:13:22.025208 3020 log.cc:1196] T 27a469ffc17447ed86860e991ff3534a P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/27a469ffc17447ed86860e991ff3534a
I20240503 08:13:22.025215 3012 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:22.025897 3024 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:22.026666 1589 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:22.027843 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet 27a469ffc17447ed86860e991ff3534a (table rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]) successfully deleted
I20240503 08:13:22.029290 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9151dcb76174b27991fdcf3839db0a5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:22.031123 31992 catalog_manager.cc:5484] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:22.031162 32141 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:22.036675 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet d9151dcb76174b27991fdcf3839db0a5 with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:22.038409 3249 tablet_replica.cc:330] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:22.038653 3249 raft_consensus.cc:2229] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:22.038975 3249 pending_rounds.cc:62] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: Trying to abort 1 pending ops.
I20240503 08:13:22.039237 3249 pending_rounds.cc:68] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023509512073056256 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d9151dcb76174b27991fdcf3839db0a5" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } } }
I20240503 08:13:22.039963 3249 raft_consensus.cc:2894] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 08:13:22.040112 3249 raft_consensus.cc:2866] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:13:22.040235 3249 raft_consensus.cc:2258] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:22.041582 3249 ts_tablet_manager.cc:1875] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:22.046604 3249 ts_tablet_manager.cc:1888] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:22.046708 3249 log.cc:1196] T d9151dcb76174b27991fdcf3839db0a5 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/d9151dcb76174b27991fdcf3839db0a5
I20240503 08:13:22.047397 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet d9151dcb76174b27991fdcf3839db0a5 (table rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]) successfully deleted
I20240503 08:13:22.088762 2328 raft_consensus.cc:1050] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:22.089588 2328 consensus_queue.cc:237] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:22.091781 32530 raft_consensus.cc:1261] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:22.091868 1989 raft_consensus.cc:1261] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:22.092329 32270 raft_consensus.cc:1261] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:22.092468 2328 consensus_queue.cc:1035] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:22.092787 3077 consensus_queue.cc:1035] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:22.093714 2328 consensus_queue.cc:1035] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:22.102468 2328 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.102964 32270 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.111153 1989 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.111733 31994 catalog_manager.cc:5484] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:22.114401 32532 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.116444 32141 consensus_queue.cc:237] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:22.122190 32272 raft_consensus.cc:1261] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:22.122414 1988 raft_consensus.cc:1261] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:22.124421 3077 consensus_queue.cc:1035] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:22.126543 3077 consensus_queue.cc:1035] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:22.127040 3077 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.127467 1988 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.130162 32272 raft_consensus.cc:2932] T 10646e4785b047d8a3ac6af5ddb06474 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.134003 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10646e4785b047d8a3ac6af5ddb06474 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:22.136193 31996 catalog_manager.cc:5484] T 10646e4785b047d8a3ac6af5ddb06474 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:22.141734 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet 10646e4785b047d8a3ac6af5ddb06474 with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:22.142073 3249 tablet_replica.cc:330] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:22.142256 3249 raft_consensus.cc:2229] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:22.142478 3249 raft_consensus.cc:2258] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:22.143330 3249 ts_tablet_manager.cc:1875] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:22.148844 3249 ts_tablet_manager.cc:1888] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:22.149025 3249 log.cc:1196] T 10646e4785b047d8a3ac6af5ddb06474 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/10646e4785b047d8a3ac6af5ddb06474
I20240503 08:13:22.149713 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet 10646e4785b047d8a3ac6af5ddb06474 (table rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]) successfully deleted
I20240503 08:13:22.187086 3310 ts_tablet_manager.cc:902] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:22.188355 3310 tablet_copy_client.cc:308] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:22.205368 3012 raft_consensus.cc:1050] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:22.206161 3012 consensus_queue.cc:237] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:22.210909 32401 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:22.210928 1725 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:22.211277 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet 86b7e012f1544986a9460cd5dac5cece from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:44321)
I20240503 08:13:22.211433 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet 86b7e012f1544986a9460cd5dac5cece from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:44321: session id = f7f6592d13d245508261964ca6ffc5c4-86b7e012f1544986a9460cd5dac5cece
I20240503 08:13:22.211684 3024 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:22.212247 3012 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:22.214270 32161 tablet_copy_source_session.cc:205] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.215752 32532 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:22.216298 3012 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:22.217279 3310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b7e012f1544986a9460cd5dac5cece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.222469 3310 tablet_copy_client.cc:783] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.222689 3310 tablet_copy_client.cc:647] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.224176 3314 tablet_copy_client.cc:1002] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.226907 3024 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:22.228199 32532 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:22.233323 32401 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:22.232165 31996 catalog_manager.cc:5484] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:22.235922 1725 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:22.243816 3335 ts_tablet_manager.cc:902] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:22.245632 3335 tablet_copy_client.cc:308] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:22.246253 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet 7d3ee53044cd43debe6b59a2ef51c230 from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:56869)
I20240503 08:13:22.246415 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet 7d3ee53044cd43debe6b59a2ef51c230 from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:56869: session id = c6b280e8ecd54855b7b21b58a223149a-7d3ee53044cd43debe6b59a2ef51c230
I20240503 08:13:22.248572 32161 tablet_copy_source_session.cc:205] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.249608 3335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3ee53044cd43debe6b59a2ef51c230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.252604 32141 consensus_queue.cc:237] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:22.254159 3335 tablet_copy_client.cc:783] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.254375 3335 tablet_copy_client.cc:647] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
W20240503 08:13:22.255137 32074 consensus_peers.cc:485] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b44429c4434d47c090f9857389c70702. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:22.255193 32272 raft_consensus.cc:1261] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:22.255237 32532 raft_consensus.cc:1261] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:22.255733 3076 consensus_queue.cc:1035] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.256052 2328 consensus_queue.cc:1035] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.257845 3340 tablet_copy_client.cc:1002] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.259080 3076 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:22.259685 32532 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:22.260712 32272 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:22.264765 31994 catalog_manager.cc:5484] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:22.277249 32272 consensus_queue.cc:237] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:22.279502 32401 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:22.279582 32207 consensus_peers.cc:485] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4c5c3fc4dc146d4818d6fdfbd4ac695. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:22.279922 32141 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:22.279994 2042 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.280470 2042 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.282306 3012 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:22.283077 32401 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:22.283468 32141 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:22.288136 31996 catalog_manager.cc:5484] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index -1 to 4, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:22.301239 32272 consensus_queue.cc:237] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:22.303335 32401 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:22.303540 32532 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:22.303665 32206 consensus_peers.cc:485] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: feee2677e29f4614825042f6fb663389. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:22.303958 3024 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.304209 1906 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.306799 2042 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:22.307394 32401 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:22.307493 32532 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:22.311995 31992 catalog_manager.cc:5484] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:22.315395 3360 ts_tablet_manager.cc:902] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:22.326143 3360 tablet_copy_client.cc:308] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:22.326851 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet 130b43c1e784400fb8851a71428014a1 from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:37127)
I20240503 08:13:22.327028 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet 130b43c1e784400fb8851a71428014a1 from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:37127: session id = f7f6592d13d245508261964ca6ffc5c4-130b43c1e784400fb8851a71428014a1
I20240503 08:13:22.329496 32292 tablet_copy_source_session.cc:205] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.330591 3360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130b43c1e784400fb8851a71428014a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.335307 3360 tablet_copy_client.cc:783] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.335557 3360 tablet_copy_client.cc:647] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.336882 3366 tablet_copy_client.cc:1002] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.337476 32401 consensus_queue.cc:237] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:22.339462 32532 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:22.339733 32337 consensus_peers.cc:485] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4bd617fecd1483e944365252edf852c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:22.339777 32141 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:22.339952 3086 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.340226 2391 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.343916 3086 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:22.344342 32141 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:22.345909 32139 consensus_queue.cc:237] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:22.347610 1988 raft_consensus.cc:1261] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:22.348057 32530 raft_consensus.cc:1261] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:22.348163 3076 consensus_queue.cc:1035] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:22.348624 2328 consensus_queue.cc:1035] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:22.349944 31992 catalog_manager.cc:5484] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:22.350703 2328 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.351101 1988 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.353608 32530 raft_consensus.cc:2932] T b7996d398e3440ebbb8f942bda8f6833 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.356098 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b7996d398e3440ebbb8f942bda8f6833 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:22.357822 31992 catalog_manager.cc:5484] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 reported cstate change: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:22.362987 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet b7996d398e3440ebbb8f942bda8f6833 with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:22.362983 32532 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:22.374002 3373 tablet_replica.cc:330] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:22.374208 3379 ts_tablet_manager.cc:902] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:22.374202 3373 raft_consensus.cc:2229] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:22.374538 3373 raft_consensus.cc:2258] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:22.375445 3373 ts_tablet_manager.cc:1875] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:22.380869 32141 consensus_queue.cc:237] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:22.382359 3373 ts_tablet_manager.cc:1888] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:22.382534 3373 log.cc:1196] T b7996d398e3440ebbb8f942bda8f6833 P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/b7996d398e3440ebbb8f942bda8f6833
I20240503 08:13:22.383370 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet b7996d398e3440ebbb8f942bda8f6833 (table rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]) successfully deleted
W20240503 08:13:22.385020 32076 consensus_peers.cc:485] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7ed99e3dd124de09353f6ed509f3eb1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:22.385179 32401 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:22.385180 32532 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:22.385799 3076 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.386057 2328 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:22.388298 2328 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:22.390324 3379 tablet_copy_client.cc:308] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:34237
I20240503 08:13:22.391156 32401 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:22.395308 32000 catalog_manager.cc:5484] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:22.402357 32422 tablet_copy_service.cc:140] P 71ad6bf07682497aaa1e9429320aef79: Received BeginTabletCopySession request for tablet e4e25ba167a248ce883fbafc4f6f5fd6 from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:49641)
I20240503 08:13:22.402524 32422 tablet_copy_service.cc:161] P 71ad6bf07682497aaa1e9429320aef79: Beginning new tablet copy session on tablet e4e25ba167a248ce883fbafc4f6f5fd6 from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:49641: session id = e615bd1588964e2fbe14f14f4aae0a98-e4e25ba167a248ce883fbafc4f6f5fd6
I20240503 08:13:22.404528 32422 tablet_copy_source_session.cc:205] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.401062 32532 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:22.405671 3379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e25ba167a248ce883fbafc4f6f5fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.409556 3393 ts_tablet_manager.cc:902] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:22.411551 3379 tablet_copy_client.cc:783] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.411763 3379 tablet_copy_client.cc:647] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.413239 3386 tablet_copy_client.cc:1002] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.417330 3393 tablet_copy_client.cc:308] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:22.417964 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet 59a99f630f59472ca5c6778525e79fbe from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:52495)
I20240503 08:13:22.418143 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet 59a99f630f59472ca5c6778525e79fbe from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:52495: session id = 6c93e47973e047d0a162a50f72f9a757-59a99f630f59472ca5c6778525e79fbe
I20240503 08:13:22.420507 32552 tablet_copy_source_session.cc:205] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.421705 3393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a99f630f59472ca5c6778525e79fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.426672 3393 tablet_copy_client.cc:783] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.427639 3393 tablet_copy_client.cc:647] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.429112 3397 tablet_copy_client.cc:1002] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.475133 3429 ts_tablet_manager.cc:902] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:22.476536 3429 tablet_copy_client.cc:308] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:22.477126 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet c970023069d44323a2b38629da1e2fad from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:52495)
I20240503 08:13:22.477284 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet c970023069d44323a2b38629da1e2fad from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:52495: session id = 6c93e47973e047d0a162a50f72f9a757-c970023069d44323a2b38629da1e2fad
I20240503 08:13:22.479462 32552 tablet_copy_source_session.cc:205] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.480612 3429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c970023069d44323a2b38629da1e2fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.485925 3429 tablet_copy_client.cc:783] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.486155 3429 tablet_copy_client.cc:647] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.487622 3433 tablet_copy_client.cc:1002] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.526243 32272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831"
dest_uuid: "272c59640332489181cbf2cc52d266bb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42558
I20240503 08:13:22.526443 32272 raft_consensus.cc:595] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:22.651486 1906 raft_consensus.cc:979] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: : Instructing follower 6c93e47973e047d0a162a50f72f9a757 to start an election
I20240503 08:13:22.651664 1906 raft_consensus.cc:1067] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Signalling peer 6c93e47973e047d0a162a50f72f9a757 to start an election
I20240503 08:13:22.652421 1725 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831"
dest_uuid: "6c93e47973e047d0a162a50f72f9a757"
from {username='slave'} at 127.24.123.2:38939
I20240503 08:13:22.652714 1725 raft_consensus.cc:482] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:22.652896 1725 raft_consensus.cc:3037] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:22.654888 1725 raft_consensus.cc:504] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:22.656936 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831" candidate_uuid: "6c93e47973e047d0a162a50f72f9a757" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:22.657130 32532 raft_consensus.cc:3037] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:22.659144 32532 raft_consensus.cc:2454] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c93e47973e047d0a162a50f72f9a757 in term 2.
I20240503 08:13:22.660149 1725 leader_election.cc:288] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [CANDIDATE]: Term 2 election: Requested vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:22.660990 32401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831" candidate_uuid: "6c93e47973e047d0a162a50f72f9a757" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:22.661201 32401 raft_consensus.cc:3037] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:22.661710 32272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831" candidate_uuid: "6c93e47973e047d0a162a50f72f9a757" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:22.661914 32272 raft_consensus.cc:3032] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:22.661985 32272 raft_consensus.cc:729] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:22.662276 32272 consensus_queue.cc:260] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:22.662690 32272 raft_consensus.cc:3037] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:22.663336 32401 raft_consensus.cc:2454] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c93e47973e047d0a162a50f72f9a757 in term 2.
I20240503 08:13:22.664005 1657 leader_election.cc:302] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 6c93e47973e047d0a162a50f72f9a757, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:22.664402 3330 raft_consensus.cc:2781] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:22.666483 32272 raft_consensus.cc:2454] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c93e47973e047d0a162a50f72f9a757 in term 2.
I20240503 08:13:22.666749 3330 raft_consensus.cc:686] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 2 LEADER]: Becoming Leader. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Running, Role: LEADER
I20240503 08:13:22.667259 3330 consensus_queue.cc:237] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:22.670666 32000 catalog_manager.cc:5484] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: term changed from 1 to 2, leader changed from 272c59640332489181cbf2cc52d266bb (127.24.123.2) to 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6). New cstate: current_term: 2 leader_uuid: "6c93e47973e047d0a162a50f72f9a757" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:22.681514 32272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e6a81a3850e49edb08c8060a45b6831"
dest_uuid: "272c59640332489181cbf2cc52d266bb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42602
I20240503 08:13:22.681695 32272 raft_consensus.cc:595] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 08:13:22.681779 32272 raft_consensus.cc:603] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 08:13:22.686440 3536 ts_tablet_manager.cc:902] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:22.687680 3536 tablet_copy_client.cc:308] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:22.688428 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet b44429c4434d47c090f9857389c70702 from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:56869)
I20240503 08:13:22.688565 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet b44429c4434d47c090f9857389c70702 from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:56869: session id = c6b280e8ecd54855b7b21b58a223149a-b44429c4434d47c090f9857389c70702
I20240503 08:13:22.690385 32161 tablet_copy_source_session.cc:205] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.691566 3536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44429c4434d47c090f9857389c70702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.695967 3536 tablet_copy_client.cc:783] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.696169 3536 tablet_copy_client.cc:647] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.697420 3545 tablet_copy_client.cc:1002] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.886677 3548 ts_tablet_manager.cc:902] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:22.887816 3548 tablet_copy_client.cc:308] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:22.888522 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:37127)
I20240503 08:13:22.888679 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:37127: session id = f7f6592d13d245508261964ca6ffc5c4-a4c5c3fc4dc146d4818d6fdfbd4ac695
I20240503 08:13:22.890548 32292 tablet_copy_source_session.cc:205] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.891664 3548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.896008 3548 tablet_copy_client.cc:783] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.896235 3548 tablet_copy_client.cc:647] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.897622 3552 tablet_copy_client.cc:1002] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.917028 3568 ts_tablet_manager.cc:902] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:22.918895 3568 tablet_copy_client.cc:308] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:22.919585 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet b7ed99e3dd124de09353f6ed509f3eb1 from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:46063)
I20240503 08:13:22.919742 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet b7ed99e3dd124de09353f6ed509f3eb1 from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:46063: session id = e615bd1588964e2fbe14f14f4aae0a98-b7ed99e3dd124de09353f6ed509f3eb1
I20240503 08:13:22.921926 32161 tablet_copy_source_session.cc:205] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.923077 3568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ed99e3dd124de09353f6ed509f3eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.928292 3568 tablet_copy_client.cc:783] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.928524 3568 tablet_copy_client.cc:647] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.929579 3578 ts_tablet_manager.cc:902] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:22.930416 3572 tablet_copy_client.cc:1002] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.932500 3578 tablet_copy_client.cc:308] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:22.933324 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet feee2677e29f4614825042f6fb663389 from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:41801)
I20240503 08:13:22.933466 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet feee2677e29f4614825042f6fb663389 from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:41801: session id = 6c93e47973e047d0a162a50f72f9a757-feee2677e29f4614825042f6fb663389
I20240503 08:13:22.935781 32292 tablet_copy_source_session.cc:205] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.937219 3578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feee2677e29f4614825042f6fb663389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.943524 3578 tablet_copy_client.cc:783] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.943756 3578 tablet_copy_client.cc:647] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.945331 3582 tablet_copy_client.cc:1002] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:22.961279 3597 ts_tablet_manager.cc:902] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:22.975905 3597 tablet_copy_client.cc:308] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:34237
I20240503 08:13:22.976644 32422 tablet_copy_service.cc:140] P 71ad6bf07682497aaa1e9429320aef79: Received BeginTabletCopySession request for tablet f4bd617fecd1483e944365252edf852c from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:49641)
I20240503 08:13:22.976804 32422 tablet_copy_service.cc:161] P 71ad6bf07682497aaa1e9429320aef79: Beginning new tablet copy session on tablet f4bd617fecd1483e944365252edf852c from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:49641: session id = e615bd1588964e2fbe14f14f4aae0a98-f4bd617fecd1483e944365252edf852c
I20240503 08:13:22.979485 32422 tablet_copy_source_session.cc:205] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:22.980579 3597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bd617fecd1483e944365252edf852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:22.985543 3597 tablet_copy_client.cc:783] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:22.985774 3597 tablet_copy_client.cc:647] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:22.987394 3604 tablet_copy_client.cc:1002] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:23.135910 32400 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:23.136538 3330 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:23.187723 32272 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:23.188243 3330 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:23.204560 32532 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:23.205302 3330 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:23.724552 3310 tablet_copy_client.cc:523] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:23.727694 3310 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:23.730854 1725 consensus_queue.cc:237] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:23.733130 32400 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:13:23.733170 32532 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:13:23.733691 3761 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:23.733983 3760 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:23.737352 3636 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:23.737928 32532 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:23.741444 32400 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:23.744482 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8e6a81a3850e49edb08c8060a45b6831 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:23.745338 31994 catalog_manager.cc:5484] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 2 leader_uuid: "6c93e47973e047d0a162a50f72f9a757" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:23.757879 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6a81a3850e49edb08c8060a45b6831 with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:23.758821 3335 tablet_copy_client.cc:523] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:23.762408 3768 tablet_replica.cc:330] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:23.762639 3768 raft_consensus.cc:2229] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:23.762900 3768 raft_consensus.cc:2258] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:23.764385 3335 tablet_bootstrap.cc:492] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:23.764406 3310 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:23.764753 3310 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:23.764981 3310 ts_tablet_manager.cc:1367] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.037s user 0.033s sys 0.000s
I20240503 08:13:23.765599 3310 raft_consensus.cc:348] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:23.765794 3310 raft_consensus.cc:729] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:23.765974 3310 consensus_queue.cc:260] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:23.766721 3310 ts_tablet_manager.cc:1398] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:23.767467 3768 ts_tablet_manager.cc:1875] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:23.767469 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-86b7e012f1544986a9460cd5dac5cece received from {username='slave'} at 127.24.123.7:44321
I20240503 08:13:23.767589 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-86b7e012f1544986a9460cd5dac5cece on tablet 86b7e012f1544986a9460cd5dac5cece with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:23.773468 3768 ts_tablet_manager.cc:1888] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:13:23.773613 3768 log.cc:1196] T 8e6a81a3850e49edb08c8060a45b6831 P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/8e6a81a3850e49edb08c8060a45b6831
I20240503 08:13:23.774353 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet 8e6a81a3850e49edb08c8060a45b6831 (table rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]) successfully deleted
I20240503 08:13:23.796643 3335 tablet_bootstrap.cc:492] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:23.797011 3335 tablet_bootstrap.cc:492] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:23.797255 3335 ts_tablet_manager.cc:1367] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.000s
I20240503 08:13:23.797922 3335 raft_consensus.cc:348] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:23.798138 3335 raft_consensus.cc:729] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:23.798326 3335 consensus_queue.cc:260] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:23.799075 3335 ts_tablet_manager.cc:1398] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:23.800040 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-7d3ee53044cd43debe6b59a2ef51c230 received from {username='slave'} at 127.24.123.5:56869
I20240503 08:13:23.800141 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-7d3ee53044cd43debe6b59a2ef51c230 on tablet 7d3ee53044cd43debe6b59a2ef51c230 with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:23.837221 3360 tablet_copy_client.cc:523] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:23.840135 3360 tablet_bootstrap.cc:492] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:23.875203 3360 tablet_bootstrap.cc:492] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:23.875676 3360 tablet_bootstrap.cc:492] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:23.875949 3360 ts_tablet_manager.cc:1367] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.036s user 0.033s sys 0.002s
I20240503 08:13:23.876610 3360 raft_consensus.cc:348] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:23.876821 3360 raft_consensus.cc:729] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:23.876996 3360 consensus_queue.cc:260] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:23.877456 3360 ts_tablet_manager.cc:1398] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:23.878108 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-130b43c1e784400fb8851a71428014a1 received from {username='slave'} at 127.24.123.7:37127
I20240503 08:13:23.878233 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-130b43c1e784400fb8851a71428014a1 on tablet 130b43c1e784400fb8851a71428014a1 with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:23.913610 3379 tablet_copy_client.cc:523] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:23.916721 3379 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:23.929534 3393 tablet_copy_client.cc:523] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:23.932647 3393 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
I20240503 08:13:23.955781 3379 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:23.956282 3379 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:23.956624 3379 ts_tablet_manager.cc:1367] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.040s user 0.032s sys 0.008s
I20240503 08:13:23.957475 3379 raft_consensus.cc:348] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:23.957800 3379 raft_consensus.cc:729] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:23.958041 3379 consensus_queue.cc:260] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:23.958935 3379 ts_tablet_manager.cc:1398] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:23.960208 32422 tablet_copy_service.cc:342] P 71ad6bf07682497aaa1e9429320aef79: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-e4e25ba167a248ce883fbafc4f6f5fd6 received from {username='slave'} at 127.24.123.8:49641
I20240503 08:13:23.960355 32422 tablet_copy_service.cc:433] P 71ad6bf07682497aaa1e9429320aef79: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-e4e25ba167a248ce883fbafc4f6f5fd6 on tablet e4e25ba167a248ce883fbafc4f6f5fd6 with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:23.971556 3393 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:23.971962 3393 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:23.972311 3393 ts_tablet_manager.cc:1367] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.040s user 0.029s sys 0.009s
I20240503 08:13:23.972961 3393 raft_consensus.cc:348] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:23.973166 3393 raft_consensus.cc:729] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:23.973363 3393 consensus_queue.cc:260] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:23.973826 3393 ts_tablet_manager.cc:1398] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:23.974571 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-59a99f630f59472ca5c6778525e79fbe received from {username='slave'} at 127.24.123.6:52495
I20240503 08:13:23.974717 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-59a99f630f59472ca5c6778525e79fbe on tablet 59a99f630f59472ca5c6778525e79fbe with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:23.988139 3429 tablet_copy_client.cc:523] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:23.991052 3429 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
I20240503 08:13:24.027750 3429 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:24.028122 3429 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:24.028400 3429 ts_tablet_manager.cc:1367] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.037s user 0.036s sys 0.000s
I20240503 08:13:24.029188 3429 raft_consensus.cc:348] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:24.029451 3429 raft_consensus.cc:729] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:24.029662 3429 consensus_queue.cc:260] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:24.030185 3429 ts_tablet_manager.cc:1398] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:24.030961 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-c970023069d44323a2b38629da1e2fad received from {username='slave'} at 127.24.123.6:52495
I20240503 08:13:24.031105 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-c970023069d44323a2b38629da1e2fad on tablet c970023069d44323a2b38629da1e2fad with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:24.139031 1725 raft_consensus.cc:1203] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.156270 1589 raft_consensus.cc:1203] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.197772 3536 tablet_copy_client.cc:523] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:24.201339 3536 tablet_bootstrap.cc:492] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:24.211912 1856 raft_consensus.cc:1203] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.229200 1856 raft_consensus.cc:1203] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.236652 3536 tablet_bootstrap.cc:492] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:24.237003 3536 tablet_bootstrap.cc:492] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:24.237232 3536 ts_tablet_manager.cc:1367] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.008s
I20240503 08:13:24.237857 3536 raft_consensus.cc:348] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:24.238054 3536 raft_consensus.cc:729] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:24.238271 3536 consensus_queue.cc:260] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:24.238828 3536 ts_tablet_manager.cc:1398] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:24.239782 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-b44429c4434d47c090f9857389c70702 received from {username='slave'} at 127.24.123.5:56869
I20240503 08:13:24.239904 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-b44429c4434d47c090f9857389c70702 on tablet b44429c4434d47c090f9857389c70702 with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:24.260555 1988 raft_consensus.cc:1203] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.388175 1725 raft_consensus.cc:1203] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.397955 3548 tablet_copy_client.cc:523] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:24.400401 3548 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:24.428494 3548 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:24.428857 3548 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:24.429080 3548 ts_tablet_manager.cc:1367] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.008s
I20240503 08:13:24.429766 3548 raft_consensus.cc:348] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:24.429996 3548 raft_consensus.cc:729] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:24.430222 3548 consensus_queue.cc:260] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:24.430749 3568 tablet_copy_client.cc:523] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:24.433662 3568 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:24.439028 3548 ts_tablet_manager.cc:1398] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240503 08:13:24.439771 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-a4c5c3fc4dc146d4818d6fdfbd4ac695 received from {username='slave'} at 127.24.123.7:37127
I20240503 08:13:24.439942 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-a4c5c3fc4dc146d4818d6fdfbd4ac695 on tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:24.440629 1856 raft_consensus.cc:1203] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.445818 3578 tablet_copy_client.cc:523] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:24.448544 3578 tablet_bootstrap.cc:492] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
I20240503 08:13:24.466439 3568 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:24.466861 3568 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:24.467161 3568 ts_tablet_manager.cc:1367] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.004s
I20240503 08:13:24.467957 3568 raft_consensus.cc:348] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:24.468151 3568 raft_consensus.cc:729] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:24.468334 3568 consensus_queue.cc:260] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:24.469435 3568 ts_tablet_manager.cc:1398] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:24.470932 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-b7ed99e3dd124de09353f6ed509f3eb1 received from {username='slave'} at 127.24.123.8:46063
I20240503 08:13:24.471050 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-b7ed99e3dd124de09353f6ed509f3eb1 on tablet b7ed99e3dd124de09353f6ed509f3eb1 with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:24.475164 1988 raft_consensus.cc:1203] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.480538 3578 tablet_bootstrap.cc:492] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:24.480866 3578 tablet_bootstrap.cc:492] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:24.481091 3578 ts_tablet_manager.cc:1367] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.033s user 0.029s sys 0.001s
I20240503 08:13:24.481693 3578 raft_consensus.cc:348] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:24.481889 3578 raft_consensus.cc:729] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:24.482065 3578 consensus_queue.cc:260] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:24.482543 3578 ts_tablet_manager.cc:1398] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:24.483371 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-feee2677e29f4614825042f6fb663389 received from {username='slave'} at 127.24.123.6:41801
I20240503 08:13:24.483557 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-feee2677e29f4614825042f6fb663389 on tablet feee2677e29f4614825042f6fb663389 with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:24.487704 3597 tablet_copy_client.cc:523] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:24.490303 3597 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:24.520560 3597 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:24.520882 3597 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:24.521108 3597 ts_tablet_manager.cc:1367] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.003s
I20240503 08:13:24.521739 3597 raft_consensus.cc:348] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:24.521924 3597 raft_consensus.cc:729] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:24.522083 3597 consensus_queue.cc:260] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:24.522522 3597 ts_tablet_manager.cc:1398] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:24.523180 32422 tablet_copy_service.cc:342] P 71ad6bf07682497aaa1e9429320aef79: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-f4bd617fecd1483e944365252edf852c received from {username='slave'} at 127.24.123.8:49641
I20240503 08:13:24.523314 32422 tablet_copy_service.cc:433] P 71ad6bf07682497aaa1e9429320aef79: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-f4bd617fecd1483e944365252edf852c on tablet f4bd617fecd1483e944365252edf852c with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:24.551434 1988 raft_consensus.cc:1203] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.602339 1725 raft_consensus.cc:1203] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.606844 2042 raft_consensus.cc:1050] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:24.607582 2042 consensus_queue.cc:237] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:24.609977 32532 raft_consensus.cc:1261] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.610069 32400 raft_consensus.cc:1261] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.610493 3976 consensus_queue.cc:1035] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.610965 1856 raft_consensus.cc:1261] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.611030 3976 consensus_queue.cc:1035] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.611515 1906 consensus_queue.cc:1035] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.614009 1906 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.617797 32400 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.622711 32000 catalog_manager.cc:5484] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.630090 32532 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.636632 1857 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.662524 3995 raft_consensus.cc:1050] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:24.668267 3995 consensus_queue.cc:237] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:24.671100 32400 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.671166 1725 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.671772 3546 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.672366 3546 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.672524 32141 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.676230 4000 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.696583 2331 raft_consensus.cc:1050] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:24.697331 2331 consensus_queue.cc:237] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:24.699224 32270 consensus_queue.cc:237] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:24.701591 1856 raft_consensus.cc:1261] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.702173 2042 consensus_queue.cc:1035] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.702741 32401 raft_consensus.cc:1261] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.705155 1589 raft_consensus.cc:1261] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.706859 2042 consensus_queue.cc:1035] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.708796 4012 consensus_queue.cc:1035] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.711231 3999 raft_consensus.cc:1050] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:24.712010 3999 consensus_queue.cc:237] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:24.714058 3997 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.717746 32402 raft_consensus.cc:1261] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.719074 2328 consensus_queue.cc:1035] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.721307 3976 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.723021 1856 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.724190 32141 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.724784 32401 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.725586 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 130b43c1e784400fb8851a71428014a1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:24.726574 1724 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.727809 31992 catalog_manager.cc:5484] T 130b43c1e784400fb8851a71428014a1 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:24.732141 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet 130b43c1e784400fb8851a71428014a1 with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:24.733263 31992 catalog_manager.cc:5484] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:24.739820 32532 consensus_queue.cc:237] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:24.745862 32270 raft_consensus.cc:1261] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.749790 1988 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.751165 32532 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.751729 4021 tablet_replica.cc:330] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:24.751942 32270 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.752228 4021 raft_consensus.cc:2229] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:24.752605 4021 raft_consensus.cc:2258] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:24.752506 4011 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.753029 1724 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.753075 4011 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.753741 4000 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.754760 3858 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.756819 4000 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.762650 1724 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.765436 4021 ts_tablet_manager.cc:1875] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:24.767283 1589 raft_consensus.cc:1203] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:24.769693 2331 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:24.770179 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c970023069d44323a2b38629da1e2fad with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:24.772730 31996 catalog_manager.cc:5484] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:24.776204 1591 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:24.777115 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet c970023069d44323a2b38629da1e2fad with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:24.784027 4021 ts_tablet_manager.cc:1888] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:24.786216 4021 log.cc:1196] T 130b43c1e784400fb8851a71428014a1 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/130b43c1e784400fb8851a71428014a1
I20240503 08:13:24.787236 4028 tablet_replica.cc:330] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:24.787566 4028 raft_consensus.cc:2229] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:24.787375 32402 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:24.788136 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet 130b43c1e784400fb8851a71428014a1 (table rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]) successfully deleted
I20240503 08:13:24.788538 4028 raft_consensus.cc:2258] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:24.789727 32401 raft_consensus.cc:2932] T 130b43c1e784400fb8851a71428014a1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.795405 4028 ts_tablet_manager.cc:1875] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:24.795238 4011 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.797052 32139 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.797892 3997 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.799377 32139 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.799516 1989 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.802902 32270 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.802498 31996 catalog_manager.cc:5484] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:24.805202 31994 catalog_manager.cc:5484] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 reported cstate change: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.806569 32532 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.820928 4012 consensus_queue.cc:1035] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.821676 4013 raft_consensus.cc:1050] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:24.822554 4013 consensus_queue.cc:237] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:24.825117 32400 consensus_queue.cc:237] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:24.828500 1988 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.829244 3993 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.832032 3993 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.832747 4028 ts_tablet_manager.cc:1888] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:24.832885 4028 log.cc:1196] T c970023069d44323a2b38629da1e2fad P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/c970023069d44323a2b38629da1e2fad
I20240503 08:13:24.834067 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet c970023069d44323a2b38629da1e2fad (table rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]) successfully deleted
I20240503 08:13:24.834035 32270 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:24.836802 1988 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.838089 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4e25ba167a248ce883fbafc4f6f5fd6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:24.841513 32532 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.841583 1856 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.842049 4012 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.842392 32400 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.843062 32270 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.843453 4012 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.843966 3858 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.846400 4013 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.846661 31992 catalog_manager.cc:5484] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:24.848810 32270 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.851243 32141 consensus_queue.cc:237] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:24.852720 4017 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.855732 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e25ba167a248ce883fbafc4f6f5fd6 with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:24.856065 4021 tablet_replica.cc:330] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:24.856261 4021 raft_consensus.cc:2229] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:24.856489 4021 raft_consensus.cc:2258] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:24.857692 1589 raft_consensus.cc:1261] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.857878 32401 raft_consensus.cc:1261] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.857692 32532 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.857714 1856 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.858323 4019 consensus_queue.cc:1035] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.858311 32402 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.860649 4019 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:24.863212 4021 ts_tablet_manager.cc:1875] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:24.864266 31992 catalog_manager.cc:5484] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:24.866629 1589 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:24.869349 4021 ts_tablet_manager.cc:1888] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:24.869462 4021 log.cc:1196] T e4e25ba167a248ce883fbafc4f6f5fd6 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/e4e25ba167a248ce883fbafc4f6f5fd6
I20240503 08:13:24.870329 4013 consensus_queue.cc:1035] T 7d3ee53044cd43debe6b59a2ef51c230 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.872067 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet e4e25ba167a248ce883fbafc4f6f5fd6 (table rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]) successfully deleted
I20240503 08:13:24.871773 32401 raft_consensus.cc:2932] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:24.875201 32141 consensus_queue.cc:237] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:24.877079 1856 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.877103 32401 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.877619 4013 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.878093 4012 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.880326 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d3ee53044cd43debe6b59a2ef51c230 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:24.881234 4020 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.882001 32401 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.882341 31996 catalog_manager.cc:5484] T 7d3ee53044cd43debe6b59a2ef51c230 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:24.884835 1856 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.888267 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86b7e012f1544986a9460cd5dac5cece with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:24.888689 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3ee53044cd43debe6b59a2ef51c230 with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:24.890435 32000 catalog_manager.cc:5484] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 reported cstate change: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:24.894176 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b7e012f1544986a9460cd5dac5cece with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:24.895196 4021 tablet_replica.cc:330] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:24.895423 4021 raft_consensus.cc:2229] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:24.895673 4021 raft_consensus.cc:2258] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:24.896425 4021 ts_tablet_manager.cc:1875] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:24.900494 4044 tablet_replica.cc:330] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:24.900700 4044 raft_consensus.cc:2229] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:24.901377 4044 raft_consensus.cc:2258] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:24.902287 4017 raft_consensus.cc:1050] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:24.902431 4021 ts_tablet_manager.cc:1888] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:24.902624 4021 log.cc:1196] T 86b7e012f1544986a9460cd5dac5cece P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/86b7e012f1544986a9460cd5dac5cece
I20240503 08:13:24.902777 4044 ts_tablet_manager.cc:1875] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:24.903115 4017 consensus_queue.cc:237] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:24.903645 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet 86b7e012f1544986a9460cd5dac5cece (table rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]) successfully deleted
I20240503 08:13:24.905306 1988 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.905354 32401 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.905548 32532 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.905880 4019 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.906807 4019 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.907151 2331 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.912379 4044 ts_tablet_manager.cc:1888] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:24.912590 4044 log.cc:1196] T 7d3ee53044cd43debe6b59a2ef51c230 P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/7d3ee53044cd43debe6b59a2ef51c230
I20240503 08:13:24.915057 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet 7d3ee53044cd43debe6b59a2ef51c230 (table rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]) successfully deleted
I20240503 08:13:24.921762 4020 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.922497 32532 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.922477 1988 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.926528 32401 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.928113 3546 raft_consensus.cc:1050] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:24.928911 3546 consensus_queue.cc:237] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:24.932193 31994 catalog_manager.cc:5484] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.935819 32401 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.937552 4000 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.939043 1724 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.939867 32270 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.940163 3997 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.940448 3995 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.942212 32141 consensus_queue.cc:237] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:24.944988 32532 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.945580 4020 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.948920 1988 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:24.949446 4020 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:24.950018 4018 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.950799 1988 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.954599 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b7ed99e3dd124de09353f6ed509f3eb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:24.955587 32530 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:24.961367 31992 catalog_manager.cc:5484] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:24.961448 3997 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.966564 1725 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.967242 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ed99e3dd124de09353f6ed509f3eb1 with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:24.967586 4028 tablet_replica.cc:330] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:24.967793 4028 raft_consensus.cc:2229] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:24.968102 4028 raft_consensus.cc:2258] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:24.970005 4028 ts_tablet_manager.cc:1875] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:24.972076 31992 catalog_manager.cc:5484] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.973762 32401 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:24.976632 4028 ts_tablet_manager.cc:1888] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:24.976801 4028 log.cc:1196] T b7ed99e3dd124de09353f6ed509f3eb1 P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/b7ed99e3dd124de09353f6ed509f3eb1
I20240503 08:13:24.977634 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet b7ed99e3dd124de09353f6ed509f3eb1 (table rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]) successfully deleted
I20240503 08:13:24.983116 3982 raft_consensus.cc:1050] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:24.984030 3982 consensus_queue.cc:237] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:24.989240 3858 raft_consensus.cc:1050] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:24.990026 3858 consensus_queue.cc:237] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:24.993921 1989 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.994488 32141 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.994489 3999 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.995077 3993 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:24.997643 32530 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:24.998183 3858 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.007963 3993 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:25.012037 1989 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:25.012938 1856 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.012904 31992 catalog_manager.cc:5484] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:25.013680 2042 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.017756 32401 consensus_queue.cc:237] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:25.018771 32141 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:25.022200 32139 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:25.022580 1989 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:25.022755 3993 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:25.024735 3993 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:25.027411 32139 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:25.030061 3999 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:25.031416 1989 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:25.032701 31994 catalog_manager.cc:5484] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 5 to 6, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:25.033331 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4bd617fecd1483e944365252edf852c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:25.034205 32401 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.034824 3982 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.035005 32530 consensus_queue.cc:237] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:25.037647 32141 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.036494 32270 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:25.038890 3982 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.040482 1906 raft_consensus.cc:1050] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:25.040678 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet f4bd617fecd1483e944365252edf852c with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:25.041040 4021 tablet_replica.cc:330] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:25.041232 4021 raft_consensus.cc:2229] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:25.041395 4021 pending_rounds.cc:62] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: Trying to abort 1 pending ops.
I20240503 08:13:25.041455 4021 pending_rounds.cc:68] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023509524437176320 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f4bd617fecd1483e944365252edf852c" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } } }
I20240503 08:13:25.041694 1906 consensus_queue.cc:237] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:25.041831 4021 raft_consensus.cc:2894] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 08:13:25.041930 4021 raft_consensus.cc:2866] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:13:25.042038 4021 raft_consensus.cc:2258] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:25.046459 4021 ts_tablet_manager.cc:1875] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:25.056025 4021 ts_tablet_manager.cc:1888] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:25.056159 4021 log.cc:1196] T f4bd617fecd1483e944365252edf852c P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/f4bd617fecd1483e944365252edf852c
I20240503 08:13:25.056735 2042 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:25.058266 32270 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:25.058382 32400 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:25.058764 3995 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:25.060925 1724 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:25.062076 32401 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.061801 3995 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:25.062047 32141 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:25.062585 3976 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.065284 31992 catalog_manager.cc:5484] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:25.065629 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59a99f630f59472ca5c6778525e79fbe with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:25.066380 1856 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:25.067456 32272 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:25.069016 4000 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:25.069844 1724 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.070699 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet f4bd617fecd1483e944365252edf852c (table rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]) successfully deleted
I20240503 08:13:25.071424 3982 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.072034 1724 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:25.076119 31991 catalog_manager.cc:5484] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:25.078084 32530 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.079587 3976 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.080133 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a99f630f59472ca5c6778525e79fbe with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:25.080488 4028 tablet_replica.cc:330] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:25.080715 4028 raft_consensus.cc:2229] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:25.080947 4028 raft_consensus.cc:2258] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:25.082214 4028 ts_tablet_manager.cc:1875] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:25.086165 3976 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:25.089126 32270 consensus_queue.cc:237] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:25.090991 32530 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:25.094156 1856 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:25.094164 32141 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:25.094796 3982 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:25.095899 1906 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:25.096153 32401 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:25.098055 31992 catalog_manager.cc:5484] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:25.099601 32139 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:25.098856 3982 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:25.104436 1725 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:25.111245 31996 catalog_manager.cc:5484] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:25.113654 1856 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:25.116461 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:25.123009 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:25.127111 4028 ts_tablet_manager.cc:1888] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:25.127581 4028 log.cc:1196] T 59a99f630f59472ca5c6778525e79fbe P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/59a99f630f59472ca5c6778525e79fbe
I20240503 08:13:25.132212 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet 59a99f630f59472ca5c6778525e79fbe (table rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]) successfully deleted
I20240503 08:13:25.132889 4028 tablet_replica.cc:330] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:25.133141 4028 raft_consensus.cc:2229] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:25.133414 4028 raft_consensus.cc:2258] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:25.135150 4028 ts_tablet_manager.cc:1875] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:25.144119 4028 ts_tablet_manager.cc:1888] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:25.144241 4028 log.cc:1196] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/a4c5c3fc4dc146d4818d6fdfbd4ac695
I20240503 08:13:25.145267 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 (table rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]) successfully deleted
I20240503 08:13:25.165016 4017 raft_consensus.cc:1050] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:25.165722 4017 consensus_queue.cc:237] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:25.168293 32530 raft_consensus.cc:1261] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.168339 32270 raft_consensus.cc:1261] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.168984 4018 consensus_queue.cc:1035] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.169467 2331 consensus_queue.cc:1035] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.169697 1589 raft_consensus.cc:1261] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:25.170271 2331 consensus_queue.cc:1035] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:25.172536 4018 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:25.173699 1589 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:25.175379 32530 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:25.179713 31996 catalog_manager.cc:5484] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:25.179682 32270 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:25.184865 32139 consensus_queue.cc:237] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:25.186581 1589 raft_consensus.cc:1261] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:25.186863 32530 raft_consensus.cc:1261] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:25.187407 2331 consensus_queue.cc:1035] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:25.188586 2331 consensus_queue.cc:1035] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:25.191274 1589 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:25.190804 2331 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:25.196270 32532 raft_consensus.cc:2932] T b44429c4434d47c090f9857389c70702 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:25.196974 31991 catalog_manager.cc:5484] T b44429c4434d47c090f9857389c70702 P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 5 to 6, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:25.199088 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b44429c4434d47c090f9857389c70702 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:25.200888 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet b44429c4434d47c090f9857389c70702 with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:25.201632 4044 tablet_replica.cc:330] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:25.201817 4044 raft_consensus.cc:2229] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:25.201977 4044 raft_consensus.cc:2258] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:25.202854 4044 ts_tablet_manager.cc:1875] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:25.209277 4044 ts_tablet_manager.cc:1888] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:25.209411 4044 log.cc:1196] T b44429c4434d47c090f9857389c70702 P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/b44429c4434d47c090f9857389c70702
I20240503 08:13:25.210136 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet b44429c4434d47c090f9857389c70702 (table rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]) successfully deleted
I20240503 08:13:25.257246 32270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "feee2677e29f4614825042f6fb663389"
dest_uuid: "272c59640332489181cbf2cc52d266bb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42798
I20240503 08:13:25.257449 32270 raft_consensus.cc:595] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:25.282320 32270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "feee2677e29f4614825042f6fb663389"
dest_uuid: "272c59640332489181cbf2cc52d266bb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42818
I20240503 08:13:25.282524 32270 raft_consensus.cc:595] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:25.298373 32270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "feee2677e29f4614825042f6fb663389"
dest_uuid: "272c59640332489181cbf2cc52d266bb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42832
I20240503 08:13:25.298553 32270 raft_consensus.cc:595] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:25.510239 1906 raft_consensus.cc:979] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: : Instructing follower 09a050703a1a4c0a8dfc04e4c2853797 to start an election
I20240503 08:13:25.510434 1906 raft_consensus.cc:1067] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Signalling peer 09a050703a1a4c0a8dfc04e4c2853797 to start an election
I20240503 08:13:25.511118 32532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "feee2677e29f4614825042f6fb663389"
dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
from {username='slave'} at 127.24.123.2:53159
I20240503 08:13:25.511411 32532 raft_consensus.cc:482] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:25.511560 32532 raft_consensus.cc:3037] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:25.514748 32532 raft_consensus.cc:504] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:25.516108 32401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee2677e29f4614825042f6fb663389" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:25.516347 32401 raft_consensus.cc:3037] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:25.516422 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee2677e29f4614825042f6fb663389" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:25.516662 32270 raft_consensus.cc:3032] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:25.516729 32270 raft_consensus.cc:729] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:25.516992 32270 consensus_queue.cc:260] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:25.517436 32270 raft_consensus.cc:3037] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:25.518549 32401 raft_consensus.cc:2454] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 2.
I20240503 08:13:25.519423 1725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feee2677e29f4614825042f6fb663389" candidate_uuid: "09a050703a1a4c0a8dfc04e4c2853797" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6c93e47973e047d0a162a50f72f9a757"
I20240503 08:13:25.519672 1725 raft_consensus.cc:3037] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:25.520366 32270 raft_consensus.cc:2454] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 2.
I20240503 08:13:25.520866 32466 leader_election.cc:302] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:25.515735 32532 leader_election.cc:288] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [CANDIDATE]: Term 2 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447)
I20240503 08:13:25.521184 3995 raft_consensus.cc:2781] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:25.521358 3995 raft_consensus.cc:686] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 LEADER]: Becoming Leader. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Running, Role: LEADER
I20240503 08:13:25.521617 3995 consensus_queue.cc:237] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:25.522114 1725 raft_consensus.cc:2454] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a050703a1a4c0a8dfc04e4c2853797 in term 2.
I20240503 08:13:25.524852 32000 catalog_manager.cc:5484] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: term changed from 1 to 2, leader changed from 272c59640332489181cbf2cc52d266bb (127.24.123.2) to 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4). New cstate: current_term: 2 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:25.528328 32270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "feee2677e29f4614825042f6fb663389"
dest_uuid: "272c59640332489181cbf2cc52d266bb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42862
I20240503 08:13:25.528493 32270 raft_consensus.cc:595] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 08:13:25.528565 32270 raft_consensus.cc:603] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 08:13:25.972749 32401 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:25.973201 4000 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:26.099906 32270 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:26.100469 4000 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:26.115468 1725 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:26.116113 4000 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:26.497663 32532 consensus_queue.cc:237] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:26.499186 1725 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:13:26.499359 32401 raft_consensus.cc:1261] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:13:26.499714 4178 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:26.500034 3995 consensus_queue.cc:1035] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:26.502364 4000 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:26.502985 32401 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:26.506314 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet feee2677e29f4614825042f6fb663389 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:26.506387 1725 raft_consensus.cc:2932] T feee2677e29f4614825042f6fb663389 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:26.507439 32000 catalog_manager.cc:5484] T feee2677e29f4614825042f6fb663389 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 5 to 7, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 2 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:26.517238 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet feee2677e29f4614825042f6fb663389 with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:26.517824 4194 tablet_replica.cc:330] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:26.518013 4194 raft_consensus.cc:2229] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:26.518193 4194 raft_consensus.cc:2258] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:26.519060 4194 ts_tablet_manager.cc:1875] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:26.523517 4194 ts_tablet_manager.cc:1888] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:13:26.523617 4194 log.cc:1196] T feee2677e29f4614825042f6fb663389 P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/feee2677e29f4614825042f6fb663389
I20240503 08:13:26.524176 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet feee2677e29f4614825042f6fb663389 (table rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]) successfully deleted
I20240503 08:13:26.745055 32510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:26.745702 2825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:26.775015 32251 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:26.780046 32119 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:26.792205 1571 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:26.796344 1969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:26.801611 32380 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:13:26.828750 1705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:26.831382 1837 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:26.939215 32270 consensus_queue.cc:237] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:26.941435 32141 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:26.942068 4247 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:26.944092 1725 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:26.944903 4247 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:26.947947 4247 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:26.948341 1725 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:26.952172 32141 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
W20240503 08:13:26.954342 32206 consensus_peers.cc:485] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9541e40695844a1b05ca1bc0dd8a10d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:26.957165 31992 catalog_manager.cc:5484] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:26.969915 32401 consensus_queue.cc:237] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:26.985129 1989 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:26.985477 32337 consensus_peers.cc:485] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4bd617fecd1483e944365252edf852c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:26.985947 3993 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:26.986085 32141 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:26.986579 3993 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:26.988699 4265 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:26.989248 1989 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:26.989984 32141 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:26.994704 31995 catalog_manager.cc:5484] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:27.005371 32532 consensus_queue.cc:237] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:27.007488 32401 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:27.008057 32465 consensus_peers.cc:485] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84bef4a86df446d0b4ff490822068fbb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.008286 3995 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:27.009615 32270 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:27.010154 3995 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:27.010793 4275 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.011407 32401 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.013162 32270 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.017486 31995 catalog_manager.cc:5484] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index -1 to 4, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.031129 32270 consensus_queue.cc:237] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:27.033187 1856 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.033200 32141 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:27.033325 32206 consensus_peers.cc:485] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4c5c3fc4dc146d4818d6fdfbd4ac695. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.033717 4264 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.033977 4247 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.036444 4254 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:27.037045 1856 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:27.040042 32141 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:27.041345 32000 catalog_manager.cc:5484] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 6 to 7, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:27.055814 32141 consensus_queue.cc:237] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:27.058228 32270 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:27.058422 32076 consensus_peers.cc:485] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bd736780ea840ab9b6630a4acd80ae9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.058434 1589 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.058964 4295 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.059247 4020 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.062171 4295 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:27.062390 32270 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:27.065414 1589 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:27.066848 31992 catalog_manager.cc:5484] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 6 to 7, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:27.085283 32532 consensus_queue.cc:237] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:27.088233 1589 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.088595 32141 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.088788 3995 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.089223 4275 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:27.095176 32466 consensus_peers.cc:485] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f4ac5dea8024445ad2cd1ca724f2290. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.101753 3995 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.102320 32141 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.110531 32000 catalog_manager.cc:5484] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:27.118825 1591 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.142858 32401 consensus_queue.cc:237] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
W20240503 08:13:27.145427 32336 consensus_peers.cc:485] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 -> Peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Couldn't send request to peer 6c93e47973e047d0a162a50f72f9a757. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4e25ba167a248ce883fbafc4f6f5fd6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.145480 32270 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.145503 1989 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.146037 4316 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.146353 3993 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.148699 4265 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:27.149251 1989 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:27.151733 32270 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } } }
I20240503 08:13:27.153501 31992 catalog_manager.cc:5484] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 6 to 7, NON_VOTER 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:27.167246 32401 consensus_queue.cc:237] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:27.169435 32532 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.169499 1725 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:27.169780 32337 consensus_peers.cc:485] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95491576997c4392b482b4627760d818. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.170053 3993 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.170356 4265 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.172744 3993 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:27.173321 1725 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:27.173671 32532 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:27.177291 32000 catalog_manager.cc:5484] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:27.191489 32270 consensus_queue.cc:237] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
W20240503 08:13:27.193776 32205 consensus_peers.cc:485] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25f1956062e84b92a3b036fc5dff4b46. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.193882 32532 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.193943 1856 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.194504 4264 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.194799 4254 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.197162 4264 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.197742 1856 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.203079 31992 catalog_manager.cc:5484] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:27.218752 32532 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.232585 32532 consensus_queue.cc:237] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:27.235445 1725 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.235502 32270 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.236200 3995 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.236709 3995 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:27.241602 32467 consensus_peers.cc:485] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Couldn't send request to peer f7f6592d13d245508261964ca6ffc5c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59a99f630f59472ca5c6778525e79fbe. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.241771 4275 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:27.245653 1725 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:27.251099 31996 catalog_manager.cc:5484] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:27.253225 32270 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } } }
I20240503 08:13:27.272317 32401 consensus_queue.cc:237] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:27.274756 1856 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.275315 3993 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.276319 32141 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.278470 3993 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.280947 3993 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.281353 1856 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
W20240503 08:13:27.283102 32336 consensus_peers.cc:485] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af5c3865ca694bf090dce19f1100e5eb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.285717 31992 catalog_manager.cc:5484] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 reported cstate change: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.287369 32141 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.307003 1725 consensus_queue.cc:237] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:27.309870 32401 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:13:27.311149 3760 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:13:27.318014 1658 consensus_peers.cc:485] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e6a81a3850e49edb08c8060a45b6831. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.319628 32532 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:13:27.319926 3760 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.323488 32401 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.329386 31996 catalog_manager.cc:5484] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 7 to 8, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 2 leader_uuid: "6c93e47973e047d0a162a50f72f9a757" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.330353 4358 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:27.335366 32532 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.351495 32141 consensus_queue.cc:237] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:27.353812 1989 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:27.354010 32074 consensus_peers.cc:485] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7ed99e3dd124de09353f6ed509f3eb1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.354408 4299 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.354605 32532 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.355118 4299 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.357779 4298 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.358384 1989 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.363823 31996 catalog_manager.cc:5484] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 reported cstate change: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.364801 32532 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:27.381716 32270 consensus_queue.cc:237] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:27.384158 32141 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.384275 1856 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:27.384807 4264 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:27.385133 32206 consensus_peers.cc:485] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 400a0317f9184c99a449f7f160fa998f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:27.385392 4254 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:27.399165 4379 ts_tablet_manager.cc:902] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:27.402501 4247 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.405476 1856 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.408063 32139 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.411883 31992 catalog_manager.cc:5484] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 reported cstate change: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:27.434860 4379 tablet_copy_client.cc:308] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:27.445139 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet f9541e40695844a1b05ca1bc0dd8a10d from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:58965)
I20240503 08:13:27.445333 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet f9541e40695844a1b05ca1bc0dd8a10d from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:58965: session id = 237c0b0fe44549a89d1b5bfc8c55da91-f9541e40695844a1b05ca1bc0dd8a10d
I20240503 08:13:27.447839 32292 tablet_copy_source_session.cc:205] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.453225 4379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9541e40695844a1b05ca1bc0dd8a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.464010 4379 tablet_copy_client.cc:783] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.464459 4379 tablet_copy_client.cc:647] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.466395 4388 tablet_copy_client.cc:1002] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.484388 4405 ts_tablet_manager.cc:902] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:27.491185 4405 tablet_copy_client.cc:308] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:34237
I20240503 08:13:27.491770 32422 tablet_copy_service.cc:140] P 71ad6bf07682497aaa1e9429320aef79: Received BeginTabletCopySession request for tablet f4bd617fecd1483e944365252edf852c from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:48777)
I20240503 08:13:27.491932 32422 tablet_copy_service.cc:161] P 71ad6bf07682497aaa1e9429320aef79: Beginning new tablet copy session on tablet f4bd617fecd1483e944365252edf852c from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:48777: session id = f7f6592d13d245508261964ca6ffc5c4-f4bd617fecd1483e944365252edf852c
I20240503 08:13:27.493793 32422 tablet_copy_source_session.cc:205] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.494911 4405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bd617fecd1483e944365252edf852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.501802 4405 tablet_copy_client.cc:783] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.502090 4405 tablet_copy_client.cc:647] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.503876 4413 tablet_copy_client.cc:1002] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.547406 4444 ts_tablet_manager.cc:902] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:27.548859 4444 tablet_copy_client.cc:308] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:27.549453 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:41801)
I20240503 08:13:27.549583 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:41801: session id = 6c93e47973e047d0a162a50f72f9a757-a4c5c3fc4dc146d4818d6fdfbd4ac695
I20240503 08:13:27.552001 32292 tablet_copy_source_session.cc:205] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.553457 4444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.560286 4444 tablet_copy_client.cc:783] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.560573 4444 tablet_copy_client.cc:647] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.562234 4448 tablet_copy_client.cc:1002] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.563962 4453 ts_tablet_manager.cc:902] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:27.576625 4453 tablet_copy_client.cc:308] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:27.577415 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet 84bef4a86df446d0b4ff490822068fbb from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:35625)
I20240503 08:13:27.577597 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet 84bef4a86df446d0b4ff490822068fbb from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:35625: session id = c6b280e8ecd54855b7b21b58a223149a-84bef4a86df446d0b4ff490822068fbb
I20240503 08:13:27.579789 32552 tablet_copy_source_session.cc:205] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.580885 4453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bef4a86df446d0b4ff490822068fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.585794 4453 tablet_copy_client.cc:783] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.585984 4453 tablet_copy_client.cc:647] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.587491 4463 tablet_copy_client.cc:1002] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.643424 4474 ts_tablet_manager.cc:902] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:27.657292 4477 ts_tablet_manager.cc:902] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: Initiating tablet copy from peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:27.668519 4478 ts_tablet_manager.cc:902] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
I20240503 08:13:27.668856 4479 ts_tablet_manager.cc:902] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:27.674589 4478 tablet_copy_client.cc:308] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:34237
I20240503 08:13:27.681712 32422 tablet_copy_service.cc:140] P 71ad6bf07682497aaa1e9429320aef79: Received BeginTabletCopySession request for tablet af5c3865ca694bf090dce19f1100e5eb from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:47343)
I20240503 08:13:27.681865 32422 tablet_copy_service.cc:161] P 71ad6bf07682497aaa1e9429320aef79: Beginning new tablet copy session on tablet af5c3865ca694bf090dce19f1100e5eb from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:47343: session id = 237c0b0fe44549a89d1b5bfc8c55da91-af5c3865ca694bf090dce19f1100e5eb
I20240503 08:13:27.683475 4477 tablet_copy_client.cc:308] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:34237
I20240503 08:13:27.684583 32421 tablet_copy_service.cc:140] P 71ad6bf07682497aaa1e9429320aef79: Received BeginTabletCopySession request for tablet e4e25ba167a248ce883fbafc4f6f5fd6 from peer 6c93e47973e047d0a162a50f72f9a757 ({username='slave'} at 127.24.123.6:47045)
I20240503 08:13:27.684756 32422 tablet_copy_source_session.cc:205] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.684787 32421 tablet_copy_service.cc:161] P 71ad6bf07682497aaa1e9429320aef79: Beginning new tablet copy session on tablet e4e25ba167a248ce883fbafc4f6f5fd6 from peer 6c93e47973e047d0a162a50f72f9a757 at {username='slave'} at 127.24.123.6:47045: session id = 6c93e47973e047d0a162a50f72f9a757-e4e25ba167a248ce883fbafc4f6f5fd6
I20240503 08:13:27.686710 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5c3865ca694bf090dce19f1100e5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.688135 32421 tablet_copy_source_session.cc:205] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.689373 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e25ba167a248ce883fbafc4f6f5fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.693054 4478 tablet_copy_client.cc:783] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.693359 4478 tablet_copy_client.cc:647] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.695089 4479 tablet_copy_client.cc:308] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:27.695144 4491 tablet_copy_client.cc:1002] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.696482 4490 ts_tablet_manager.cc:902] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:27.696563 4474 tablet_copy_client.cc:308] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:34237
I20240503 08:13:27.697505 32421 tablet_copy_service.cc:140] P 71ad6bf07682497aaa1e9429320aef79: Received BeginTabletCopySession request for tablet 95491576997c4392b482b4627760d818 from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:48777)
I20240503 08:13:27.697687 32421 tablet_copy_service.cc:161] P 71ad6bf07682497aaa1e9429320aef79: Beginning new tablet copy session on tablet 95491576997c4392b482b4627760d818 from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:48777: session id = f7f6592d13d245508261964ca6ffc5c4-95491576997c4392b482b4627760d818
I20240503 08:13:27.697846 4477 tablet_copy_client.cc:783] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.698068 4477 tablet_copy_client.cc:647] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.700773 32421 tablet_copy_source_session.cc:205] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.701925 4494 tablet_copy_client.cc:1002] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.701961 4474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95491576997c4392b482b4627760d818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.706943 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet 2f4ac5dea8024445ad2cd1ca724f2290 from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:42783)
I20240503 08:13:27.707098 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet 2f4ac5dea8024445ad2cd1ca724f2290 from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:42783: session id = 237c0b0fe44549a89d1b5bfc8c55da91-2f4ac5dea8024445ad2cd1ca724f2290
I20240503 08:13:27.709393 32552 tablet_copy_source_session.cc:205] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.709947 4474 tablet_copy_client.cc:783] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.710175 4474 tablet_copy_client.cc:647] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.711616 4492 tablet_copy_client.cc:1002] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.712219 4490 tablet_copy_client.cc:308] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:27.712889 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet 2bd736780ea840ab9b6630a4acd80ae9 from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:46063)
I20240503 08:13:27.713019 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet 2bd736780ea840ab9b6630a4acd80ae9 from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:46063: session id = e615bd1588964e2fbe14f14f4aae0a98-2bd736780ea840ab9b6630a4acd80ae9
I20240503 08:13:27.715090 32161 tablet_copy_source_session.cc:205] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.715957 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4ac5dea8024445ad2cd1ca724f2290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.716270 4490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd736780ea840ab9b6630a4acd80ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.721002 4479 tablet_copy_client.cc:783] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.721271 4479 tablet_copy_client.cc:647] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.722709 4493 tablet_copy_client.cc:1002] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.722954 4490 tablet_copy_client.cc:783] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.723170 4490 tablet_copy_client.cc:647] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.724923 4498 tablet_copy_client.cc:1002] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.740131 4499 ts_tablet_manager.cc:902] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:27.743577 4499 tablet_copy_client.cc:308] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:27.748710 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet 59a99f630f59472ca5c6778525e79fbe from peer f7f6592d13d245508261964ca6ffc5c4 ({username='slave'} at 127.24.123.7:60785)
I20240503 08:13:27.748873 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet 59a99f630f59472ca5c6778525e79fbe from peer f7f6592d13d245508261964ca6ffc5c4 at {username='slave'} at 127.24.123.7:60785: session id = f7f6592d13d245508261964ca6ffc5c4-59a99f630f59472ca5c6778525e79fbe
I20240503 08:13:27.750831 32552 tablet_copy_source_session.cc:205] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.751983 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a99f630f59472ca5c6778525e79fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.756484 4499 tablet_copy_client.cc:783] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.756695 4499 tablet_copy_client.cc:647] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.758159 4503 tablet_copy_client.cc:1002] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.790534 4505 ts_tablet_manager.cc:902] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:27.791885 4505 tablet_copy_client.cc:308] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:27.792542 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet 25f1956062e84b92a3b036fc5dff4b46 from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:49993)
I20240503 08:13:27.792711 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet 25f1956062e84b92a3b036fc5dff4b46 from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:49993: session id = c6b280e8ecd54855b7b21b58a223149a-25f1956062e84b92a3b036fc5dff4b46
I20240503 08:13:27.794512 32292 tablet_copy_source_session.cc:205] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.795547 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f1956062e84b92a3b036fc5dff4b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.799813 4505 tablet_copy_client.cc:783] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.800067 4505 tablet_copy_client.cc:647] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.800841 4510 ts_tablet_manager.cc:902] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:27.801607 4509 tablet_copy_client.cc:1002] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.802337 4510 tablet_copy_client.cc:308] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:27.802964 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet b7ed99e3dd124de09353f6ed509f3eb1 from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:56869)
I20240503 08:13:27.803138 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet b7ed99e3dd124de09353f6ed509f3eb1 from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:56869: session id = c6b280e8ecd54855b7b21b58a223149a-b7ed99e3dd124de09353f6ed509f3eb1
I20240503 08:13:27.805166 32161 tablet_copy_source_session.cc:205] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.806288 4510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ed99e3dd124de09353f6ed509f3eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.811818 4510 tablet_copy_client.cc:783] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.812112 4510 tablet_copy_client.cc:647] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.813828 4514 tablet_copy_client.cc:1002] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.885859 4565 ts_tablet_manager.cc:902] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447)
I20240503 08:13:27.895294 4565 tablet_copy_client.cc:308] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.6:36447
I20240503 08:13:27.895463 4569 ts_tablet_manager.cc:902] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:27.898439 4569 tablet_copy_client.cc:308] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:27.899118 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet 400a0317f9184c99a449f7f160fa998f from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:58965)
I20240503 08:13:27.899252 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet 400a0317f9184c99a449f7f160fa998f from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:58965: session id = 237c0b0fe44549a89d1b5bfc8c55da91-400a0317f9184c99a449f7f160fa998f
I20240503 08:13:27.901840 32292 tablet_copy_source_session.cc:205] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.902024 1745 tablet_copy_service.cc:140] P 6c93e47973e047d0a162a50f72f9a757: Received BeginTabletCopySession request for tablet 8e6a81a3850e49edb08c8060a45b6831 from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:58571)
I20240503 08:13:27.902240 1745 tablet_copy_service.cc:161] P 6c93e47973e047d0a162a50f72f9a757: Beginning new tablet copy session on tablet 8e6a81a3850e49edb08c8060a45b6831 from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:58571: session id = 237c0b0fe44549a89d1b5bfc8c55da91-8e6a81a3850e49edb08c8060a45b6831
I20240503 08:13:27.904693 4569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400a0317f9184c99a449f7f160fa998f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.905555 1745 tablet_copy_source_session.cc:205] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:27.907435 4565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a81a3850e49edb08c8060a45b6831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.911437 4569 tablet_copy_client.cc:783] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.911733 4569 tablet_copy_client.cc:647] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.912570 4565 tablet_copy_client.cc:783] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:27.912760 4565 tablet_copy_client.cc:647] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:27.913594 4580 tablet_copy_client.cc:1002] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:27.914273 4570 tablet_copy_client.cc:1002] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:28.966913 4379 tablet_copy_client.cc:523] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:28.970180 4379 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:28.979220 4379 log.cc:824] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:29.004223 4405 tablet_copy_client.cc:523] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.006610 4405 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:29.010697 4379 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.011112 4379 tablet_bootstrap.cc:492] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:29.011483 4379 ts_tablet_manager.cc:1367] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.042s user 0.044s sys 0.000s
I20240503 08:13:29.016541 4379 raft_consensus.cc:348] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.016885 4379 raft_consensus.cc:729] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:29.017357 4379 consensus_queue.cc:260] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.018811 4379 ts_tablet_manager.cc:1398] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 08:13:29.020262 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-f9541e40695844a1b05ca1bc0dd8a10d received from {username='slave'} at 127.24.123.9:58965
I20240503 08:13:29.020395 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-f9541e40695844a1b05ca1bc0dd8a10d on tablet f9541e40695844a1b05ca1bc0dd8a10d with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:29.042620 4405 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.043105 4405 tablet_bootstrap.cc:492] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:29.043398 4405 ts_tablet_manager.cc:1367] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.005s
I20240503 08:13:29.044086 4405 raft_consensus.cc:348] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:29.044314 4405 raft_consensus.cc:729] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:29.044479 4405 consensus_queue.cc:260] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:29.045168 4405 ts_tablet_manager.cc:1398] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:29.045888 32421 tablet_copy_service.cc:342] P 71ad6bf07682497aaa1e9429320aef79: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-f4bd617fecd1483e944365252edf852c received from {username='slave'} at 127.24.123.7:48777
I20240503 08:13:29.046000 32421 tablet_copy_service.cc:433] P 71ad6bf07682497aaa1e9429320aef79: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-f4bd617fecd1483e944365252edf852c on tablet f4bd617fecd1483e944365252edf852c with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:29.062633 4444 tablet_copy_client.cc:523] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.065078 4444 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
I20240503 08:13:29.087867 4453 tablet_copy_client.cc:523] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.090448 4453 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:29.104233 4444 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.104755 4444 tablet_bootstrap.cc:492] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:29.105082 4444 ts_tablet_manager.cc:1367] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.040s user 0.033s sys 0.007s
I20240503 08:13:29.105839 4444 raft_consensus.cc:348] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:29.106182 4444 raft_consensus.cc:729] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:29.106432 4444 consensus_queue.cc:260] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:29.107050 4444 ts_tablet_manager.cc:1398] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:29.108753 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-a4c5c3fc4dc146d4818d6fdfbd4ac695 received from {username='slave'} at 127.24.123.6:41801
I20240503 08:13:29.108893 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-a4c5c3fc4dc146d4818d6fdfbd4ac695 on tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:29.129804 4453 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.130169 4453 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:29.130395 4453 ts_tablet_manager.cc:1367] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.040s user 0.035s sys 0.000s
I20240503 08:13:29.131145 4453 raft_consensus.cc:348] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:29.131383 4453 raft_consensus.cc:729] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:29.131587 4453 consensus_queue.cc:260] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:29.138505 4453 ts_tablet_manager.cc:1398] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.008s user 0.010s sys 0.001s
I20240503 08:13:29.139335 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-84bef4a86df446d0b4ff490822068fbb received from {username='slave'} at 127.24.123.5:35625
I20240503 08:13:29.139465 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-84bef4a86df446d0b4ff490822068fbb on tablet 84bef4a86df446d0b4ff490822068fbb with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:29.145949 1589 raft_consensus.cc:1203] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:13:29.195727 4478 tablet_copy_client.cc:523] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.198128 4478 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:29.202303 4477 tablet_copy_client.cc:523] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.205399 4477 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap starting.
I20240503 08:13:29.211956 4474 tablet_copy_client.cc:523] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.215240 4474 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:29.223044 4479 tablet_copy_client.cc:523] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.225415 4490 tablet_copy_client.cc:523] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.226759 4479 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:29.228847 4490 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:29.258591 4499 tablet_copy_client.cc:523] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.262115 4499 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: Bootstrap starting.
I20240503 08:13:29.274036 2845 raft_consensus.cc:1203] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.291507 4474 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.295300 4474 tablet_bootstrap.cc:492] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:29.295678 4474 ts_tablet_manager.cc:1367] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.081s user 0.045s sys 0.008s
I20240503 08:13:29.300642 4474 raft_consensus.cc:348] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:29.301283 4479 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.301791 4479 tablet_bootstrap.cc:492] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:29.302110 4479 ts_tablet_manager.cc:1367] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.075s user 0.050s sys 0.000s
I20240503 08:13:29.302461 4505 tablet_copy_client.cc:523] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.303216 4479 raft_consensus.cc:348] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.303548 4479 raft_consensus.cc:729] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:29.303793 4479 consensus_queue.cc:260] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.304505 4479 ts_tablet_manager.cc:1398] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:29.305063 4474 raft_consensus.cc:729] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:29.305320 4474 consensus_queue.cc:260] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:29.305759 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-2f4ac5dea8024445ad2cd1ca724f2290 received from {username='slave'} at 127.24.123.9:42783
I20240503 08:13:29.305828 4505 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:29.305902 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-2f4ac5dea8024445ad2cd1ca724f2290 on tablet 2f4ac5dea8024445ad2cd1ca724f2290 with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:29.306694 4474 ts_tablet_manager.cc:1398] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:13:29.308223 32421 tablet_copy_service.cc:342] P 71ad6bf07682497aaa1e9429320aef79: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-95491576997c4392b482b4627760d818 received from {username='slave'} at 127.24.123.7:48777
I20240503 08:13:29.308341 32421 tablet_copy_service.cc:433] P 71ad6bf07682497aaa1e9429320aef79: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-95491576997c4392b482b4627760d818 on tablet 95491576997c4392b482b4627760d818 with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:29.314154 4510 tablet_copy_client.cc:523] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.309118 4477 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.315121 4477 tablet_bootstrap.cc:492] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: Bootstrap complete.
I20240503 08:13:29.315505 4477 ts_tablet_manager.cc:1367] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: Time spent bootstrapping tablet: real 0.110s user 0.047s sys 0.011s
I20240503 08:13:29.316325 4477 raft_consensus.cc:348] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:29.317652 4510 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:29.317814 4477 raft_consensus.cc:729] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Initialized, Role: LEARNER
I20240503 08:13:29.318070 4477 consensus_queue.cc:260] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: true } }
I20240503 08:13:29.321475 4477 ts_tablet_manager.cc:1398] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:13:29.328238 32421 tablet_copy_service.cc:342] P 71ad6bf07682497aaa1e9429320aef79: Request end of tablet copy session 6c93e47973e047d0a162a50f72f9a757-e4e25ba167a248ce883fbafc4f6f5fd6 received from {username='slave'} at 127.24.123.6:47045
I20240503 08:13:29.328362 32421 tablet_copy_service.cc:433] P 71ad6bf07682497aaa1e9429320aef79: ending tablet copy session 6c93e47973e047d0a162a50f72f9a757-e4e25ba167a248ce883fbafc4f6f5fd6 on tablet e4e25ba167a248ce883fbafc4f6f5fd6 with peer 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:29.334652 4490 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.335105 4490 tablet_bootstrap.cc:492] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:29.335433 4490 ts_tablet_manager.cc:1367] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.107s user 0.057s sys 0.000s
I20240503 08:13:29.336212 4490 raft_consensus.cc:348] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:29.336514 4490 raft_consensus.cc:729] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:29.336745 4490 consensus_queue.cc:260] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:29.346896 4499 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.347404 4499 tablet_bootstrap.cc:492] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: Bootstrap complete.
I20240503 08:13:29.347395 4490 ts_tablet_manager.cc:1398] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20240503 08:13:29.348115 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-2bd736780ea840ab9b6630a4acd80ae9 received from {username='slave'} at 127.24.123.8:46063
I20240503 08:13:29.348233 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-2bd736780ea840ab9b6630a4acd80ae9 on tablet 2bd736780ea840ab9b6630a4acd80ae9 with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:29.347934 4499 ts_tablet_manager.cc:1367] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: Time spent bootstrapping tablet: real 0.086s user 0.032s sys 0.018s
I20240503 08:13:29.349413 4499 raft_consensus.cc:348] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:29.349916 4499 raft_consensus.cc:729] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Initialized, Role: LEARNER
I20240503 08:13:29.350179 4499 consensus_queue.cc:260] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: NON_VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: true } }
I20240503 08:13:29.350767 4499 ts_tablet_manager.cc:1398] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:29.351553 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session f7f6592d13d245508261964ca6ffc5c4-59a99f630f59472ca5c6778525e79fbe received from {username='slave'} at 127.24.123.7:60785
I20240503 08:13:29.351672 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session f7f6592d13d245508261964ca6ffc5c4-59a99f630f59472ca5c6778525e79fbe on tablet 59a99f630f59472ca5c6778525e79fbe with peer f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:29.362848 1856 raft_consensus.cc:1203] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.365121 4478 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.365276 1856 raft_consensus.cc:1203] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.365551 4478 tablet_bootstrap.cc:492] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:29.365815 4478 ts_tablet_manager.cc:1367] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.168s user 0.038s sys 0.010s
I20240503 08:13:29.366545 4478 raft_consensus.cc:348] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.366823 4478 raft_consensus.cc:729] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:29.367050 4478 consensus_queue.cc:260] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.373950 4478 ts_tablet_manager.cc:1398] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:13:29.376368 32421 tablet_copy_service.cc:342] P 71ad6bf07682497aaa1e9429320aef79: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-af5c3865ca694bf090dce19f1100e5eb received from {username='slave'} at 127.24.123.9:47343
I20240503 08:13:29.376489 32421 tablet_copy_service.cc:433] P 71ad6bf07682497aaa1e9429320aef79: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-af5c3865ca694bf090dce19f1100e5eb on tablet af5c3865ca694bf090dce19f1100e5eb with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:29.381548 4505 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.381989 4505 tablet_bootstrap.cc:492] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:29.382310 4505 ts_tablet_manager.cc:1367] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.077s user 0.048s sys 0.016s
I20240503 08:13:29.383226 4505 raft_consensus.cc:348] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:29.383577 4505 raft_consensus.cc:729] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:29.383810 4505 consensus_queue.cc:260] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:29.384516 4505 ts_tablet_manager.cc:1398] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:29.385473 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-25f1956062e84b92a3b036fc5dff4b46 received from {username='slave'} at 127.24.123.5:49993
I20240503 08:13:29.385588 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-25f1956062e84b92a3b036fc5dff4b46 on tablet 25f1956062e84b92a3b036fc5dff4b46 with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:29.400228 4510 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.400590 4510 tablet_bootstrap.cc:492] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:29.400820 4510 ts_tablet_manager.cc:1367] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.083s user 0.049s sys 0.002s
I20240503 08:13:29.401398 4510 raft_consensus.cc:348] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:29.401619 4510 raft_consensus.cc:729] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:29.401784 4510 consensus_queue.cc:260] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:29.402282 4510 ts_tablet_manager.cc:1398] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:29.402959 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-b7ed99e3dd124de09353f6ed509f3eb1 received from {username='slave'} at 127.24.123.5:56869
I20240503 08:13:29.403084 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-b7ed99e3dd124de09353f6ed509f3eb1 on tablet b7ed99e3dd124de09353f6ed509f3eb1 with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:29.413942 4569 tablet_copy_client.cc:523] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.414597 4565 tablet_copy_client.cc:523] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:29.417486 4569 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:29.417723 4565 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:29.442117 1589 raft_consensus.cc:1203] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.466728 4569 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.467144 4565 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:29.467144 4569 tablet_bootstrap.cc:492] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:29.467439 4569 ts_tablet_manager.cc:1367] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.050s user 0.042s sys 0.008s
I20240503 08:13:29.467499 4565 tablet_bootstrap.cc:492] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:29.467734 4565 ts_tablet_manager.cc:1367] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.050s user 0.047s sys 0.000s
I20240503 08:13:29.468194 4569 raft_consensus.cc:348] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.468506 4569 raft_consensus.cc:729] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:29.468410 4565 raft_consensus.cc:348] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.468634 4565 raft_consensus.cc:729] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:29.468669 4569 consensus_queue.cc:260] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.468827 4565 consensus_queue.cc:260] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:29.469246 4569 ts_tablet_manager.cc:1398] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:29.469383 4565 ts_tablet_manager.cc:1398] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:29.470086 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-400a0317f9184c99a449f7f160fa998f received from {username='slave'} at 127.24.123.9:58965
I20240503 08:13:29.470221 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-400a0317f9184c99a449f7f160fa998f on tablet 400a0317f9184c99a449f7f160fa998f with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:29.470203 1745 tablet_copy_service.cc:342] P 6c93e47973e047d0a162a50f72f9a757: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-8e6a81a3850e49edb08c8060a45b6831 received from {username='slave'} at 127.24.123.9:58571
I20240503 08:13:29.470369 1745 tablet_copy_service.cc:433] P 6c93e47973e047d0a162a50f72f9a757: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-8e6a81a3850e49edb08c8060a45b6831 on tablet 8e6a81a3850e49edb08c8060a45b6831 with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:29.478044 2845 raft_consensus.cc:1203] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.511446 1725 raft_consensus.cc:1203] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.527781 1989 raft_consensus.cc:1203] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.538126 1856 raft_consensus.cc:1203] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.570641 2845 raft_consensus.cc:1203] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:13:29.613217 2845 raft_consensus.cc:1203] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.689194 2845 raft_consensus.cc:1203] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.764039 4979 raft_consensus.cc:1050] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:29.764768 4979 consensus_queue.cc:237] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:29.766891 1589 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:29.767267 32270 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:29.767388 32400 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:29.767396 4981 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:29.767926 4979 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:29.768210 4601 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:29.770678 4979 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:29.771325 32400 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:29.773959 1589 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:29.775705 32000 catalog_manager.cc:5484] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:29.785794 32270 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:29.796557 4264 raft_consensus.cc:1050] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:29.797142 4264 consensus_queue.cc:237] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:29.799201 1725 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.799235 32139 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.799616 2845 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.799803 4990 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.800185 4264 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.800442 4247 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.803259 4990 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:29.803880 32139 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:29.804881 2845 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:29.807492 1725 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:29.808434 1724 raft_consensus.cc:1203] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.808887 31996 catalog_manager.cc:5484] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:29.813885 32270 consensus_queue.cc:237] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:29.816488 2845 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:29.816943 4247 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:29.817543 1725 raft_consensus.cc:1261] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:29.817971 4247 consensus_queue.cc:1035] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:29.818995 4889 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:29.819710 2845 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:29.820448 1725 raft_consensus.cc:2932] T f9541e40695844a1b05ca1bc0dd8a10d P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:29.821993 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9541e40695844a1b05ca1bc0dd8a10d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:29.823606 31996 catalog_manager.cc:5484] T f9541e40695844a1b05ca1bc0dd8a10d P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:29.827550 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet f9541e40695844a1b05ca1bc0dd8a10d with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:29.828157 4994 tablet_replica.cc:330] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:29.828338 4994 raft_consensus.cc:2229] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:29.828625 4994 raft_consensus.cc:2258] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:29.829347 4994 ts_tablet_manager.cc:1875] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:29.833657 4994 ts_tablet_manager.cc:1888] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:29.833755 4994 log.cc:1196] T f9541e40695844a1b05ca1bc0dd8a10d P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/f9541e40695844a1b05ca1bc0dd8a10d
I20240503 08:13:29.834354 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet f9541e40695844a1b05ca1bc0dd8a10d (table rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]) successfully deleted
I20240503 08:13:29.936491 4996 raft_consensus.cc:1050] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:29.937215 4996 consensus_queue.cc:237] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:29.939549 1989 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.939535 32139 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.940025 1857 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.940708 4996 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.941318 4979 raft_consensus.cc:1050] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:29.942021 4979 consensus_queue.cc:237] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:29.944141 1856 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.944309 1725 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.944424 4998 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.944967 32270 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.945597 4981 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.946552 4316 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.946650 4981 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.947978 4981 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.953575 4264 raft_consensus.cc:1050] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:29.954512 4264 consensus_queue.cc:237] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:29.957270 32532 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.957320 1589 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.957860 4247 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.958174 5008 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.958665 1856 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:29.959280 4264 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:29.981040 4601 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:29.981621 4996 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:29.982151 4247 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:29.982494 1988 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:29.985694 32270 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:29.986071 1725 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:29.988145 32000 catalog_manager.cc:5484] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:29.988305 32532 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:29.992089 1591 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:29.994037 1589 raft_consensus.cc:1203] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:29.996202 31992 catalog_manager.cc:5484] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:29.997653 1855 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:29.999562 32139 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.006196 1856 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.008288 4998 raft_consensus.cc:1050] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:30.009155 4998 consensus_queue.cc:237] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:30.011519 1855 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:29.998847 31996 catalog_manager.cc:5484] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.017627 32270 consensus_queue.cc:237] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:30.019261 1591 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.020296 4264 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.020617 5012 raft_consensus.cc:1050] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:30.021229 5012 consensus_queue.cc:237] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:30.021530 1856 raft_consensus.cc:1261] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.023443 4264 consensus_queue.cc:1035] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.024252 4264 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.024580 1725 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.027462 4247 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.031903 1856 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.032658 32141 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.032733 32532 consensus_queue.cc:237] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:30.033286 4889 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.036258 1724 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.036602 1857 raft_consensus.cc:1261] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.037123 1856 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.037138 4981 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.037842 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25f1956062e84b92a3b036fc5dff4b46 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:30.039151 4981 consensus_queue.cc:1035] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.040692 4316 raft_consensus.cc:1050] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79: attempting to promote NON_VOTER f7f6592d13d245508261964ca6ffc5c4 to VOTER
I20240503 08:13:30.040796 5008 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.040652 1591 raft_consensus.cc:2932] T 25f1956062e84b92a3b036fc5dff4b46 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.041507 4316 consensus_queue.cc:237] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:30.038306 31992 catalog_manager.cc:5484] T 25f1956062e84b92a3b036fc5dff4b46 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.044760 4278 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.050724 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f1956062e84b92a3b036fc5dff4b46 with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:30.051676 1724 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.053220 1857 raft_consensus.cc:2932] T 59a99f630f59472ca5c6778525e79fbe P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.056784 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59a99f630f59472ca5c6778525e79fbe with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:30.057473 1855 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4 [term 1 LEARNER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.058030 4996 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.058211 31996 catalog_manager.cc:5484] T 59a99f630f59472ca5c6778525e79fbe P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 8 to 9, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.062124 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a99f630f59472ca5c6778525e79fbe with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:30.062567 32532 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.062810 5025 tablet_replica.cc:330] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:30.062860 1724 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.063016 5025 raft_consensus.cc:2229] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.063194 4996 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.063349 5025 raft_consensus.cc:2258] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.063480 4995 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.064182 5025 ts_tablet_manager.cc:1875] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.070590 5025 ts_tablet_manager.cc:1888] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:30.070785 5025 log.cc:1196] T 59a99f630f59472ca5c6778525e79fbe P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/59a99f630f59472ca5c6778525e79fbe
I20240503 08:13:30.072232 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet 59a99f630f59472ca5c6778525e79fbe (table rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]) successfully deleted
I20240503 08:13:30.077950 5023 tablet_replica.cc:330] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:30.078770 5023 raft_consensus.cc:2229] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.079916 5023 raft_consensus.cc:2258] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.080155 2845 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.080797 5023 ts_tablet_manager.cc:1875] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.085201 4996 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.089401 5023 ts_tablet_manager.cc:1888] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:30.089560 5023 log.cc:1196] T 25f1956062e84b92a3b036fc5dff4b46 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/25f1956062e84b92a3b036fc5dff4b46
I20240503 08:13:30.090385 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet 25f1956062e84b92a3b036fc5dff4b46 (table rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]) successfully deleted
I20240503 08:13:30.090277 5008 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:30.093611 32141 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:30.095000 1856 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:30.096768 32139 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.097352 4316 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.099260 1855 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.100219 4316 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.100428 32000 catalog_manager.cc:5484] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 reported cstate change: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:30.113178 4995 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.116678 4979 raft_consensus.cc:1050] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:30.116534 1723 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.117422 4979 consensus_queue.cc:237] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:30.118499 1854 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.121229 2844 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.121802 1591 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.122511 4278 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.123998 32000 catalog_manager.cc:5484] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.124320 4601 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.129300 32400 consensus_queue.cc:237] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:30.131245 32141 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.131742 1855 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.131871 4979 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.132571 4998 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.134646 32532 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.143579 4979 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.144848 4316 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.148636 2843 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.149175 32140 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.149892 31992 catalog_manager.cc:5484] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.152402 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95491576997c4392b482b4627760d818 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:30.152427 1589 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.154284 1854 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.154637 31995 catalog_manager.cc:5484] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.158916 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet 95491576997c4392b482b4627760d818 with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:30.160928 4998 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.163901 2843 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.164710 32530 consensus_queue.cc:237] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:30.167572 2842 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.169088 4979 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.171563 32140 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.173688 31992 catalog_manager.cc:5484] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.174160 1854 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.181138 32400 consensus_queue.cc:237] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:30.183229 1591 raft_consensus.cc:1261] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.183933 4979 consensus_queue.cc:1035] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.184677 2841 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.185256 5008 raft_consensus.cc:1050] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:30.185252 4996 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.186054 5008 consensus_queue.cc:237] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:30.186789 4278 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.193459 1854 raft_consensus.cc:1261] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.196792 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f4ac5dea8024445ad2cd1ca724f2290 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:30.198693 32000 catalog_manager.cc:5484] T 2f4ac5dea8024445ad2cd1ca724f2290 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:30.200364 4316 consensus_queue.cc:1035] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.203249 1591 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.203347 4996 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.207677 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af5c3865ca694bf090dce19f1100e5eb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:30.210045 31992 catalog_manager.cc:5484] T af5c3865ca694bf090dce19f1100e5eb P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.212626 2841 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.215963 2844 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.216466 5008 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.220225 5041 raft_consensus.cc:1050] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:30.220976 5041 consensus_queue.cc:237] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:30.221316 1854 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.222271 5008 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.226311 1857 raft_consensus.cc:2932] T af5c3865ca694bf090dce19f1100e5eb P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.226931 2842 raft_consensus.cc:2932] T 2f4ac5dea8024445ad2cd1ca724f2290 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
W20240503 08:13:30.235420 32074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:34237, user_credentials={real_user=slave}} blocked reactor thread for 52422.5us
I20240503 08:13:30.236385 32140 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.236682 5023 tablet_replica.cc:330] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:30.236903 5023 raft_consensus.cc:2229] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.236874 4264 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.237138 5023 raft_consensus.cc:2258] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.237815 1989 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.238371 5042 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.239364 1725 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:30.240474 5023 ts_tablet_manager.cc:1875] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.246606 5023 ts_tablet_manager.cc:1888] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:30.246804 5023 log.cc:1196] T 95491576997c4392b482b4627760d818 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/95491576997c4392b482b4627760d818
I20240503 08:13:30.247476 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet 95491576997c4392b482b4627760d818 (table rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]) successfully deleted
I20240503 08:13:30.252106 32270 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.251964 4264 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.252923 1724 raft_consensus.cc:1261] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.253350 5039 raft_consensus.cc:1050] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:30.253782 4998 consensus_queue.cc:1035] T 95491576997c4392b482b4627760d818 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.255266 5043 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.255223 1724 raft_consensus.cc:2932] T 95491576997c4392b482b4627760d818 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:30.257014 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4ac5dea8024445ad2cd1ca724f2290 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:30.257256 1589 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.257324 4994 tablet_replica.cc:330] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:30.257503 4994 raft_consensus.cc:2229] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.257709 4994 raft_consensus.cc:2258] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.258106 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet af5c3865ca694bf090dce19f1100e5eb with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:30.258386 2844 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.259119 4994 ts_tablet_manager.cc:1875] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.259567 5038 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.263470 1854 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.263810 31995 catalog_manager.cc:5484] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 reported cstate change: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.269551 32140 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.271164 32272 consensus_queue.cc:237] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:30.272778 4994 ts_tablet_manager.cc:1888] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:30.272851 5039 consensus_queue.cc:237] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:30.273461 4994 log.cc:1196] T 2f4ac5dea8024445ad2cd1ca724f2290 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/2f4ac5dea8024445ad2cd1ca724f2290
I20240503 08:13:30.274345 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet 2f4ac5dea8024445ad2cd1ca724f2290 (table rebalance_test_table_9 [id=cb9ce5ae7cdf4227a329bf181f5d9bb6]) successfully deleted
I20240503 08:13:30.276698 4994 tablet_replica.cc:330] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:30.277668 2844 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.278527 1854 raft_consensus.cc:1261] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.278609 4994 raft_consensus.cc:2229] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.279099 4994 raft_consensus.cc:2258] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.279302 5008 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.280187 4264 consensus_queue.cc:1035] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.281579 5012 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.284732 4994 ts_tablet_manager.cc:1875] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.286370 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 400a0317f9184c99a449f7f160fa998f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:30.287761 2841 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.292980 31992 catalog_manager.cc:5484] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 reported cstate change: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.294291 32400 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:30.295513 2844 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 LEARNER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:30.295681 32532 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:30.297297 4822 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20240503 08:13:30.297375 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet 400a0317f9184c99a449f7f160fa998f with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:30.297678 5047 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.298417 5048 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.304615 1854 raft_consensus.cc:2932] T 400a0317f9184c99a449f7f160fa998f P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.309127 5038 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:30.309676 1591 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:30.311777 32272 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:30.311815 4822 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.312975 2841 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.316973 32532 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.317803 1989 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:30.320199 32000 catalog_manager.cc:5484] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 7 to 8, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:30.320982 31992 catalog_manager.cc:5484] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91 reported cstate change: config changed from index 8 to 9, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6c93e47973e047d0a162a50f72f9a757" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.328773 4994 ts_tablet_manager.cc:1888] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:30.329511 4994 log.cc:1196] T af5c3865ca694bf090dce19f1100e5eb P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/af5c3865ca694bf090dce19f1100e5eb
I20240503 08:13:30.330411 32400 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.331243 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet af5c3865ca694bf090dce19f1100e5eb (table rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]) successfully deleted
I20240503 08:13:30.331476 1724 consensus_queue.cc:237] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:30.335184 32532 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:30.335395 2841 raft_consensus.cc:1261] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:30.336118 5039 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:30.337523 32140 consensus_queue.cc:237] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:30.338264 4822 consensus_queue.cc:1035] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:30.339270 5039 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.339665 1989 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.340232 1589 raft_consensus.cc:1261] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.340176 5042 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.340739 5042 consensus_queue.cc:1035] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.343111 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8e6a81a3850e49edb08c8060a45b6831 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:30.342836 5043 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:30.343814 1589 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:30.346599 31991 catalog_manager.cc:5484] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 9 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 2 leader_uuid: "6c93e47973e047d0a162a50f72f9a757" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.347499 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2bd736780ea840ab9b6630a4acd80ae9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:30.347687 1989 raft_consensus.cc:2932] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:30.348273 2844 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.349301 32532 raft_consensus.cc:2932] T 8e6a81a3850e49edb08c8060a45b6831 P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:30.350412 32000 catalog_manager.cc:5484] T 2bd736780ea840ab9b6630a4acd80ae9 P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 8 to 9, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:30.353529 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6a81a3850e49edb08c8060a45b6831 with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:30.353940 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bd736780ea840ab9b6630a4acd80ae9 with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:30.354250 5025 tablet_replica.cc:330] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:30.354414 5064 tablet_replica.cc:330] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:30.354446 5025 raft_consensus.cc:2229] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.354622 5064 raft_consensus.cc:2229] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.354667 5025 raft_consensus.cc:2258] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.355299 5064 raft_consensus.cc:2258] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.356984 5025 ts_tablet_manager.cc:1875] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.357196 5064 ts_tablet_manager.cc:1875] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.357339 4994 tablet_replica.cc:330] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:30.357527 4994 raft_consensus.cc:2229] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.358390 4994 raft_consensus.cc:2258] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.359555 4994 ts_tablet_manager.cc:1875] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.363502 5064 ts_tablet_manager.cc:1888] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:30.363664 5064 log.cc:1196] T 8e6a81a3850e49edb08c8060a45b6831 P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/8e6a81a3850e49edb08c8060a45b6831
I20240503 08:13:30.364311 5025 ts_tablet_manager.cc:1888] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:30.364423 5025 log.cc:1196] T 2bd736780ea840ab9b6630a4acd80ae9 P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/2bd736780ea840ab9b6630a4acd80ae9
I20240503 08:13:30.364413 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet 8e6a81a3850e49edb08c8060a45b6831 (table rebalance_test_table_0 [id=a0ca0dd6c11140d6b13ade456f2e060c]) successfully deleted
I20240503 08:13:30.365096 4994 ts_tablet_manager.cc:1888] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:30.365077 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet 2bd736780ea840ab9b6630a4acd80ae9 (table rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]) successfully deleted
I20240503 08:13:30.365214 4994 log.cc:1196] T 400a0317f9184c99a449f7f160fa998f P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/400a0317f9184c99a449f7f160fa998f
I20240503 08:13:30.365902 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet 400a0317f9184c99a449f7f160fa998f (table rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]) successfully deleted
I20240503 08:13:30.368376 4995 raft_consensus.cc:1050] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: attempting to promote NON_VOTER 6c93e47973e047d0a162a50f72f9a757 to VOTER
I20240503 08:13:30.369200 4995 consensus_queue.cc:237] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:30.372296 1989 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.372393 1724 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 1 LEARNER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.373068 32270 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 71ad6bf07682497aaa1e9429320aef79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.372994 4998 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.373454 4316 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.374779 4316 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.377224 4998 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:30.380565 1989 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:30.381156 32270 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:30.382074 32000 catalog_manager.cc:5484] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "71ad6bf07682497aaa1e9429320aef79" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:30.384284 1724 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:30.430675 32400 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4e25ba167a248ce883fbafc4f6f5fd6"
dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46638
I20240503 08:13:30.430984 32400 raft_consensus.cc:595] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:30.474184 32270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4c5c3fc4dc146d4818d6fdfbd4ac695"
dest_uuid: "272c59640332489181cbf2cc52d266bb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54330
I20240503 08:13:30.474414 32270 raft_consensus.cc:595] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:30.499207 32532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84bef4a86df446d0b4ff490822068fbb"
dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52616
I20240503 08:13:30.499464 32532 raft_consensus.cc:595] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:30.505154 5012 raft_consensus.cc:979] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: : Instructing follower f7f6592d13d245508261964ca6ffc5c4 to start an election
I20240503 08:13:30.505367 5008 raft_consensus.cc:1067] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Signalling peer f7f6592d13d245508261964ca6ffc5c4 to start an election
I20240503 08:13:30.506062 1854 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4c5c3fc4dc146d4818d6fdfbd4ac695"
dest_uuid: "f7f6592d13d245508261964ca6ffc5c4"
from {username='slave'} at 127.24.123.2:48965
I20240503 08:13:30.506358 1854 raft_consensus.cc:482] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:30.506542 1854 raft_consensus.cc:3037] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.509213 1854 raft_consensus.cc:504] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:30.511541 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5c3fc4dc146d4818d6fdfbd4ac695" candidate_uuid: "f7f6592d13d245508261964ca6ffc5c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:30.511772 32140 raft_consensus.cc:3037] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.511737 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5c3fc4dc146d4818d6fdfbd4ac695" candidate_uuid: "f7f6592d13d245508261964ca6ffc5c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:30.511948 32270 raft_consensus.cc:3032] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:30.512079 32270 raft_consensus.cc:729] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:30.512420 32270 consensus_queue.cc:260] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:30.512796 32270 raft_consensus.cc:3037] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.512152 1854 leader_election.cc:288] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [CANDIDATE]: Term 2 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447)
I20240503 08:13:30.514453 32140 raft_consensus.cc:2454] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f6592d13d245508261964ca6ffc5c4 in term 2.
I20240503 08:13:30.516754 32270 raft_consensus.cc:2454] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f6592d13d245508261964ca6ffc5c4 in term 2.
I20240503 08:13:30.517714 1790 leader_election.cc:302] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90, f7f6592d13d245508261964ca6ffc5c4; no voters:
I20240503 08:13:30.518080 4819 raft_consensus.cc:2781] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:30.518831 5042 raft_consensus.cc:1050] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:30.519658 5042 consensus_queue.cc:237] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:30.522071 1589 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.522228 1989 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.522723 5040 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.523054 5041 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.524099 32532 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:30.524724 5040 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:30.531816 1724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5c3fc4dc146d4818d6fdfbd4ac695" candidate_uuid: "f7f6592d13d245508261964ca6ffc5c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6c93e47973e047d0a162a50f72f9a757"
I20240503 08:13:30.531870 4819 raft_consensus.cc:686] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 2 LEADER]: Becoming Leader. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Running, Role: LEADER
I20240503 08:13:30.532030 1724 raft_consensus.cc:3037] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.532419 4819 consensus_queue.cc:237] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:30.534626 1724 raft_consensus.cc:2454] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f6592d13d245508261964ca6ffc5c4 in term 2.
I20240503 08:13:30.538499 31995 catalog_manager.cc:5484] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 reported cstate change: term changed from 1 to 2, leader changed from 272c59640332489181cbf2cc52d266bb (127.24.123.2) to f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7). New cstate: current_term: 2 leader_uuid: "f7f6592d13d245508261964ca6ffc5c4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:30.540038 32400 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f4bd617fecd1483e944365252edf852c"
dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46650
I20240503 08:13:30.540257 32400 raft_consensus.cc:595] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:30.551348 5042 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.552124 32532 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.552904 1988 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.556085 1589 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.557750 31991 catalog_manager.cc:5484] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.563905 32140 consensus_queue.cc:237] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:30.565760 1589 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.566449 1988 raft_consensus.cc:1261] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:30.566380 5040 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.568434 5040 consensus_queue.cc:1035] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:30.569908 5040 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.572636 1989 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.574349 1589 raft_consensus.cc:2932] T b7ed99e3dd124de09353f6ed509f3eb1 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.577790 31995 catalog_manager.cc:5484] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 reported cstate change: config changed from index 8 to 9, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:30.580956 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b7ed99e3dd124de09353f6ed509f3eb1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:30.582235 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ed99e3dd124de09353f6ed509f3eb1 with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:30.582623 5023 tablet_replica.cc:330] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:30.582813 5023 raft_consensus.cc:2229] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:30.583170 5023 raft_consensus.cc:2258] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:30.584874 5023 ts_tablet_manager.cc:1875] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:30.591075 5023 ts_tablet_manager.cc:1888] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:30.591200 5023 log.cc:1196] T b7ed99e3dd124de09353f6ed509f3eb1 P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/b7ed99e3dd124de09353f6ed509f3eb1
I20240503 08:13:30.591816 4996 raft_consensus.cc:979] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: : Instructing follower e615bd1588964e2fbe14f14f4aae0a98 to start an election
I20240503 08:13:30.591967 4996 raft_consensus.cc:1067] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Signalling peer e615bd1588964e2fbe14f14f4aae0a98 to start an election
I20240503 08:13:30.592051 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet b7ed99e3dd124de09353f6ed509f3eb1 (table rebalance_test_table_3 [id=f98283e72dfa4ba7a9730fc6d07ed34c]) successfully deleted
I20240503 08:13:30.593156 1988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f4bd617fecd1483e944365252edf852c"
dest_uuid: "e615bd1588964e2fbe14f14f4aae0a98"
from {username='slave'} at 127.24.123.3:35309
I20240503 08:13:30.593457 1988 raft_consensus.cc:482] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:30.593581 1988 raft_consensus.cc:3037] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.596030 1988 raft_consensus.cc:504] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:30.597726 1988 leader_election.cc:288] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603)
I20240503 08:13:30.598477 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bd617fecd1483e944365252edf852c" candidate_uuid: "e615bd1588964e2fbe14f14f4aae0a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:30.598656 32140 raft_consensus.cc:3037] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.599319 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bd617fecd1483e944365252edf852c" candidate_uuid: "e615bd1588964e2fbe14f14f4aae0a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:30.599526 32400 raft_consensus.cc:3032] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:30.599619 32400 raft_consensus.cc:729] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Running, Role: LEADER
I20240503 08:13:30.599931 32400 consensus_queue.cc:260] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:30.600381 32400 raft_consensus.cc:3037] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.600965 32140 raft_consensus.cc:2454] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e615bd1588964e2fbe14f14f4aae0a98 in term 2.
I20240503 08:13:30.602658 32400 raft_consensus.cc:2454] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e615bd1588964e2fbe14f14f4aae0a98 in term 2.
I20240503 08:13:30.603117 1921 leader_election.cc:302] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90, e615bd1588964e2fbe14f14f4aae0a98; no voters:
I20240503 08:13:30.607033 4890 raft_consensus.cc:2781] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:30.607560 4890 raft_consensus.cc:686] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 2 LEADER]: Becoming Leader. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Running, Role: LEADER
I20240503 08:13:30.608145 4890 consensus_queue.cc:237] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:30.610503 32532 consensus_queue.cc:237] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:30.610725 1857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bd617fecd1483e944365252edf852c" candidate_uuid: "e615bd1588964e2fbe14f14f4aae0a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f7f6592d13d245508261964ca6ffc5c4"
I20240503 08:13:30.611029 1857 raft_consensus.cc:3037] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.613016 32140 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:30.613581 1724 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:30.613729 1857 raft_consensus.cc:2454] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e615bd1588964e2fbe14f14f4aae0a98 in term 2.
I20240503 08:13:30.614735 4981 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.615360 31991 catalog_manager.cc:5484] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 reported cstate change: term changed from 1 to 2, leader changed from 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) to e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8). New cstate: current_term: 2 leader_uuid: "e615bd1588964e2fbe14f14f4aae0a98" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:30.615697 32466 consensus_peers.cc:485] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 734d2de8d2804b2c8d2f144157e12f77. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.615962 4981 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.618865 4981 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.619320 32140 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.623788 31991 catalog_manager.cc:5484] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.634619 1723 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.646198 32270 consensus_queue.cc:237] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:30.648746 32140 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:30.648780 1589 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:30.649092 32206 consensus_peers.cc:485] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9151dcb76174b27991fdcf3839db0a5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.649421 4264 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.649747 5008 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.664494 4264 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.665530 1589 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.668197 32141 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.671401 31991 catalog_manager.cc:5484] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.680167 4601 raft_consensus.cc:979] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: : Instructing follower c6b280e8ecd54855b7b21b58a223149a to start an election
I20240503 08:13:30.680387 4278 raft_consensus.cc:1067] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Signalling peer c6b280e8ecd54855b7b21b58a223149a to start an election
I20240503 08:13:30.681155 1589 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84bef4a86df446d0b4ff490822068fbb"
dest_uuid: "c6b280e8ecd54855b7b21b58a223149a"
from {username='slave'} at 127.24.123.4:59597
I20240503 08:13:30.681417 1589 raft_consensus.cc:482] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:30.681581 1589 raft_consensus.cc:3037] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.684156 1589 raft_consensus.cc:504] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:30.686219 1589 leader_election.cc:288] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [CANDIDATE]: Term 2 election: Requested vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:30.686926 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bef4a86df446d0b4ff490822068fbb" candidate_uuid: "c6b280e8ecd54855b7b21b58a223149a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09a050703a1a4c0a8dfc04e4c2853797"
I20240503 08:13:30.687096 32532 raft_consensus.cc:3032] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:30.687163 32532 raft_consensus.cc:729] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09a050703a1a4c0a8dfc04e4c2853797, State: Running, Role: LEADER
I20240503 08:13:30.687404 32532 consensus_queue.cc:260] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:30.687742 32532 raft_consensus.cc:3037] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.689446 32532 raft_consensus.cc:2454] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b280e8ecd54855b7b21b58a223149a in term 2.
I20240503 08:13:30.689788 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bef4a86df446d0b4ff490822068fbb" candidate_uuid: "c6b280e8ecd54855b7b21b58a223149a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:30.690049 32270 raft_consensus.cc:3037] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.692595 32270 raft_consensus.cc:2454] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b280e8ecd54855b7b21b58a223149a in term 2.
I20240503 08:13:30.692958 1524 leader_election.cc:302] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09a050703a1a4c0a8dfc04e4c2853797, 272c59640332489181cbf2cc52d266bb, c6b280e8ecd54855b7b21b58a223149a; no voters:
I20240503 08:13:30.693260 4829 raft_consensus.cc:2781] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:30.693709 4829 raft_consensus.cc:686] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 2 LEADER]: Becoming Leader. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Running, Role: LEADER
I20240503 08:13:30.694208 4829 consensus_queue.cc:237] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:30.697290 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bef4a86df446d0b4ff490822068fbb" candidate_uuid: "c6b280e8ecd54855b7b21b58a223149a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:30.697520 32400 raft_consensus.cc:3037] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.698711 31995 catalog_manager.cc:5484] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a reported cstate change: term changed from 1 to 2, leader changed from 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) to c6b280e8ecd54855b7b21b58a223149a (127.24.123.5). New cstate: current_term: 2 leader_uuid: "c6b280e8ecd54855b7b21b58a223149a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.699810 32400 raft_consensus.cc:2454] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b280e8ecd54855b7b21b58a223149a in term 2.
I20240503 08:13:30.702857 32270 consensus_queue.cc:237] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:30.704941 32141 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:30.704977 1589 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:30.705147 32206 consensus_peers.cc:485] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5de086cc1f0b497f87b055e745a38e0d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.705453 5008 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.705709 4264 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.708668 4264 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.711972 32141 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.712395 1589 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.713246 31995 catalog_manager.cc:5484] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:30.731369 32532 consensus_queue.cc:237] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:30.733716 1723 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:30.733811 32141 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:30.733968 32466 consensus_peers.cc:485] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c970023069d44323a2b38629da1e2fad. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:30.734459 4601 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.734776 4278 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.737416 4981 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.737978 32141 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.738550 1723 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.743459 31995 catalog_manager.cc:5484] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.781319 4995 raft_consensus.cc:979] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: : Instructing follower 6c93e47973e047d0a162a50f72f9a757 to start an election
I20240503 08:13:30.781519 4995 raft_consensus.cc:1067] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Signalling peer 6c93e47973e047d0a162a50f72f9a757 to start an election
I20240503 08:13:30.782135 1723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4e25ba167a248ce883fbafc4f6f5fd6"
dest_uuid: "6c93e47973e047d0a162a50f72f9a757"
from {username='slave'} at 127.24.123.3:41097
I20240503 08:13:30.782344 1723 raft_consensus.cc:482] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:30.782470 1723 raft_consensus.cc:3037] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.784864 1723 raft_consensus.cc:504] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:30.786381 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e25ba167a248ce883fbafc4f6f5fd6" candidate_uuid: "6c93e47973e047d0a162a50f72f9a757" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:30.786471 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e25ba167a248ce883fbafc4f6f5fd6" candidate_uuid: "6c93e47973e047d0a162a50f72f9a757" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "272c59640332489181cbf2cc52d266bb"
I20240503 08:13:30.786584 32400 raft_consensus.cc:3032] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:30.786682 32400 raft_consensus.cc:729] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71ad6bf07682497aaa1e9429320aef79, State: Running, Role: LEADER
I20240503 08:13:30.786701 32270 raft_consensus.cc:3037] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.786967 32400 consensus_queue.cc:260] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:30.787448 1723 leader_election.cc:288] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [CANDIDATE]: Term 2 election: Requested vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749)
I20240503 08:13:30.787462 32400 raft_consensus.cc:3037] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.789659 32270 raft_consensus.cc:2454] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c93e47973e047d0a162a50f72f9a757 in term 2.
I20240503 08:13:30.790280 32400 raft_consensus.cc:2454] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c93e47973e047d0a162a50f72f9a757 in term 2.
I20240503 08:13:30.790642 1657 leader_election.cc:302] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 272c59640332489181cbf2cc52d266bb, 6c93e47973e047d0a162a50f72f9a757, 71ad6bf07682497aaa1e9429320aef79; no voters:
I20240503 08:13:30.791014 5039 raft_consensus.cc:2781] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:30.791153 5039 raft_consensus.cc:686] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 2 LEADER]: Becoming Leader. State: Replica: 6c93e47973e047d0a162a50f72f9a757, State: Running, Role: LEADER
I20240503 08:13:30.791466 5039 consensus_queue.cc:237] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:30.795401 32000 catalog_manager.cc:5484] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: term changed from 1 to 2, leader changed from 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) to 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6). New cstate: current_term: 2 leader_uuid: "6c93e47973e047d0a162a50f72f9a757" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:30.802490 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e25ba167a248ce883fbafc4f6f5fd6" candidate_uuid: "6c93e47973e047d0a162a50f72f9a757" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e615bd1588964e2fbe14f14f4aae0a98"
I20240503 08:13:30.802724 1988 raft_consensus.cc:3037] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:30.805231 1988 raft_consensus.cc:2454] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c93e47973e047d0a162a50f72f9a757 in term 2.
I20240503 08:13:30.855521 32141 consensus_queue.cc:237] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:30.857806 1988 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:30.858498 5040 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.861972 5042 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.862699 1988 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.868324 32000 catalog_manager.cc:5484] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 reported cstate change: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
I20240503 08:13:30.869836 32270 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:30.872577 5042 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:30.875634 32270 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } }
W20240503 08:13:30.876978 32075 consensus_peers.cc:485] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169): Couldn't send request to peer 237c0b0fe44549a89d1b5bfc8c55da91. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cc8a3241af248578b6ce87b1e33cea9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:31.033428 32141 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:31.034032 5147 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:31.055770 1723 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:31.056257 5147 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:31.061123 32270 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:31.061858 5147 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:31.091691 1857 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 2 FOLLOWER]: Refusing update from remote peer e615bd1588964e2fbe14f14f4aae0a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:31.092293 4890 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:31.096526 1854 consensus_queue.cc:237] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:31.098301 1723 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:31.098807 32141 raft_consensus.cc:1261] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:31.099048 5307 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:31.099318 5308 consensus_queue.cc:1035] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:31.101545 5307 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:31.107081 31986 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:31.107807 1723 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 6c93e47973e047d0a162a50f72f9a757 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:31.113106 32141 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Refusing update from remote peer e615bd1588964e2fbe14f14f4aae0a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:31.113674 4890 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:31.114614 32141 raft_consensus.cc:2932] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:31.117622 31991 catalog_manager.cc:5484] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 reported cstate change: config changed from index 8 to 10, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 2 leader_uuid: "f7f6592d13d245508261964ca6ffc5c4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:31.123802 32532 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:31.124481 4829 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:31.125782 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:31.144948 5317 ts_tablet_manager.cc:902] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:31.158818 5320 ts_tablet_manager.cc:902] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:31.160240 32400 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer e615bd1588964e2fbe14f14f4aae0a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:31.161262 5316 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20240503 08:13:31.173470 5314 tablet_replica.cc:330] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:31.173995 5314 raft_consensus.cc:2229] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:31.174311 5314 raft_consensus.cc:2258] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:31.180642 5320 tablet_copy_client.cc:308] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:31.181807 1988 consensus_queue.cc:237] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:31.182251 5333 ts_tablet_manager.cc:902] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785)
I20240503 08:13:31.183773 5337 ts_tablet_manager.cc:902] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617)
I20240503 08:13:31.184044 5314 ts_tablet_manager.cc:1875] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:31.184186 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet c970023069d44323a2b38629da1e2fad from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:42783)
I20240503 08:13:31.184355 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet c970023069d44323a2b38629da1e2fad from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:42783: session id = 237c0b0fe44549a89d1b5bfc8c55da91-c970023069d44323a2b38629da1e2fad
I20240503 08:13:31.187271 32552 tablet_copy_source_session.cc:205] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.188736 5320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c970023069d44323a2b38629da1e2fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.192091 5337 tablet_copy_client.cc:308] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:31.192843 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet 5de086cc1f0b497f87b055e745a38e0d from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:58965)
I20240503 08:13:31.193022 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet 5de086cc1f0b497f87b055e745a38e0d from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:58965: session id = 237c0b0fe44549a89d1b5bfc8c55da91-5de086cc1f0b497f87b055e745a38e0d
I20240503 08:13:31.195994 32292 tablet_copy_source_session.cc:205] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.197961 5337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de086cc1f0b497f87b055e745a38e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.198450 5314 ts_tablet_manager.cc:1888] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:31.198570 5314 log.cc:1196] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4c5c3fc4dc146d4818d6fdfbd4ac695
I20240503 08:13:31.199616 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet a4c5c3fc4dc146d4818d6fdfbd4ac695 (table rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]) successfully deleted
I20240503 08:13:31.204242 1854 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 2 FOLLOWER]: Refusing update from remote peer e615bd1588964e2fbe14f14f4aae0a98: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:31.204428 32140 raft_consensus.cc:1261] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Refusing update from remote peer e615bd1588964e2fbe14f14f4aae0a98: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:31.204921 5316 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:31.205251 5318 consensus_queue.cc:1035] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:31.208868 4890 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:31.209619 5317 tablet_copy_client.cc:308] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:39617
I20240503 08:13:31.210404 32292 tablet_copy_service.cc:140] P 272c59640332489181cbf2cc52d266bb: Received BeginTabletCopySession request for tablet d9151dcb76174b27991fdcf3839db0a5 from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:58965)
I20240503 08:13:31.210546 32292 tablet_copy_service.cc:161] P 272c59640332489181cbf2cc52d266bb: Beginning new tablet copy session on tablet d9151dcb76174b27991fdcf3839db0a5 from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:58965: session id = 237c0b0fe44549a89d1b5bfc8c55da91-d9151dcb76174b27991fdcf3839db0a5
I20240503 08:13:31.212760 32140 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:31.213431 32292 tablet_copy_source_session.cc:205] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.214926 5317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9151dcb76174b27991fdcf3839db0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.214977 31986 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4bd617fecd1483e944365252edf852c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:31.216580 32000 catalog_manager.cc:5484] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 reported cstate change: config changed from index 8 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 2 leader_uuid: "e615bd1588964e2fbe14f14f4aae0a98" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:31.217764 5333 tablet_copy_client.cc:308] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:46785
I20240503 08:13:31.218195 5337 tablet_copy_client.cc:783] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.218423 5337 tablet_copy_client.cc:647] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.220078 32552 tablet_copy_service.cc:140] P 09a050703a1a4c0a8dfc04e4c2853797: Received BeginTabletCopySession request for tablet 734d2de8d2804b2c8d2f144157e12f77 from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:42783)
I20240503 08:13:31.220352 32552 tablet_copy_service.cc:161] P 09a050703a1a4c0a8dfc04e4c2853797: Beginning new tablet copy session on tablet 734d2de8d2804b2c8d2f144157e12f77 from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:42783: session id = 237c0b0fe44549a89d1b5bfc8c55da91-734d2de8d2804b2c8d2f144157e12f77
I20240503 08:13:31.223560 32552 tablet_copy_source_session.cc:205] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.223948 5317 tablet_copy_client.cc:783] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.224160 5317 tablet_copy_client.cc:647] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.224289 5341 tablet_copy_client.cc:1002] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:31.225307 1854 raft_consensus.cc:2932] T f4bd617fecd1483e944365252edf852c P f7f6592d13d245508261964ca6ffc5c4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } }
I20240503 08:13:31.225706 5344 tablet_copy_client.cc:1002] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:31.225692 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet f4bd617fecd1483e944365252edf852c with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:31.232429 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734d2de8d2804b2c8d2f144157e12f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.234910 5320 tablet_copy_client.cc:783] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.235234 5320 tablet_copy_client.cc:647] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.237874 5331 tablet_copy_client.cc:1002] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:31.246937 5333 tablet_copy_client.cc:783] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.247223 5333 tablet_copy_client.cc:647] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.248566 5347 tablet_replica.cc:330] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:31.248778 5347 raft_consensus.cc:2229] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:31.248986 5347 raft_consensus.cc:2258] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:31.249871 5345 tablet_copy_client.cc:1002] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:31.253446 5347 ts_tablet_manager.cc:1875] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:31.259485 5347 ts_tablet_manager.cc:1888] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:31.259639 5347 log.cc:1196] T f4bd617fecd1483e944365252edf852c P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4bd617fecd1483e944365252edf852c
I20240503 08:13:31.260265 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet f4bd617fecd1483e944365252edf852c (table rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]) successfully deleted
I20240503 08:13:31.262850 5353 ts_tablet_manager.cc:902] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:31.265134 32400 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:31.265736 4829 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:31.268841 32270 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:13:31.269780 4829 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:31.274060 32272 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:31.275455 5047 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:31.302834 5353 tablet_copy_client.cc:308] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:31.307754 32400 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:31.308349 5047 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:31.309849 1589 consensus_queue.cc:237] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:31.312458 32270 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:13:31.313486 5362 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:31.314010 32400 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:13:31.315053 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet 7cc8a3241af248578b6ce87b1e33cea9 from peer 237c0b0fe44549a89d1b5bfc8c55da91 ({username='slave'} at 127.24.123.9:60153)
I20240503 08:13:31.315229 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet 7cc8a3241af248578b6ce87b1e33cea9 from peer 237c0b0fe44549a89d1b5bfc8c55da91 at {username='slave'} at 127.24.123.9:60153: session id = 237c0b0fe44549a89d1b5bfc8c55da91-7cc8a3241af248578b6ce87b1e33cea9
I20240503 08:13:31.317756 32161 tablet_copy_source_session.cc:205] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:31.318122 1988 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:31.318241 4829 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:31.319325 5358 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:31.320266 5372 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:31.320650 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc8a3241af248578b6ce87b1e33cea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.322134 32270 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:31.324748 32400 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:31.328933 5353 tablet_copy_client.cc:783] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:31.329174 5353 tablet_copy_client.cc:647] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:31.330837 5363 tablet_copy_client.cc:1002] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:31.332813 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84bef4a86df446d0b4ff490822068fbb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:31.334930 31995 catalog_manager.cc:5484] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 reported cstate change: config changed from index 5 to 7, VOTER 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4) evicted. New cstate: current_term: 2 leader_uuid: "c6b280e8ecd54855b7b21b58a223149a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:31.338589 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bef4a86df446d0b4ff490822068fbb with delete_type TABLET_DATA_TOMBSTONED (TS 09a050703a1a4c0a8dfc04e4c2853797 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41074
I20240503 08:13:31.339300 5376 tablet_replica.cc:330] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: stopping tablet replica
I20240503 08:13:31.339514 5376 raft_consensus.cc:2229] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:31.339769 5376 raft_consensus.cc:2258] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:31.340647 5376 ts_tablet_manager.cc:1875] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:31.345955 5376 ts_tablet_manager.cc:1888] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:13:31.346048 5376 log.cc:1196] T 84bef4a86df446d0b4ff490822068fbb P 09a050703a1a4c0a8dfc04e4c2853797: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/84bef4a86df446d0b4ff490822068fbb
I20240503 08:13:31.346670 31985 catalog_manager.cc:4830] TS 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): tablet 84bef4a86df446d0b4ff490822068fbb (table rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]) successfully deleted
I20240503 08:13:31.829332 1723 consensus_queue.cc:237] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } }
I20240503 08:13:31.830897 1988 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:31.831035 32270 raft_consensus.cc:1261] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer 6c93e47973e047d0a162a50f72f9a757: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:31.831526 5372 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:31.831805 5047 consensus_queue.cc:1035] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:31.833902 5372 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:31.834477 32270 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:31.836546 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4e25ba167a248ce883fbafc4f6f5fd6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:31.836917 1988 raft_consensus.cc:2932] T e4e25ba167a248ce883fbafc4f6f5fd6 P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:31.840195 31991 catalog_manager.cc:5484] T e4e25ba167a248ce883fbafc4f6f5fd6 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 8 to 10, VOTER 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3) evicted. New cstate: current_term: 2 leader_uuid: "6c93e47973e047d0a162a50f72f9a757" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } }
I20240503 08:13:31.844256 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e25ba167a248ce883fbafc4f6f5fd6 with delete_type TABLET_DATA_TOMBSTONED (TS 71ad6bf07682497aaa1e9429320aef79 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51536
I20240503 08:13:31.845108 5472 tablet_replica.cc:330] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: stopping tablet replica
I20240503 08:13:31.845297 5472 raft_consensus.cc:2229] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:31.845477 5472 raft_consensus.cc:2258] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:31.846189 5472 ts_tablet_manager.cc:1875] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:31.850466 5472 ts_tablet_manager.cc:1888] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:31.850571 5472 log.cc:1196] T e4e25ba167a248ce883fbafc4f6f5fd6 P 71ad6bf07682497aaa1e9429320aef79: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/e4e25ba167a248ce883fbafc4f6f5fd6
I20240503 08:13:31.851197 31984 catalog_manager.cc:4830] TS 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): tablet e4e25ba167a248ce883fbafc4f6f5fd6 (table rebalance_test_table_4 [id=1a99af4e9f8b4944baa6bc26b7b83565]) successfully deleted
I20240503 08:13:32.724658 5337 tablet_copy_client.cc:523] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:32.726040 5317 tablet_copy_client.cc:523] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:32.728077 5337 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:32.729280 5317 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:32.738209 5320 tablet_copy_client.cc:523] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:32.741781 5320 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:32.750273 5333 tablet_copy_client.cc:523] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:32.755304 5333 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:32.793421 5337 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:32.793915 5337 tablet_bootstrap.cc:492] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:32.794279 5337 ts_tablet_manager.cc:1367] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.066s user 0.047s sys 0.013s
I20240503 08:13:32.797451 5337 raft_consensus.cc:348] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.797775 5337 raft_consensus.cc:729] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:32.799630 5337 consensus_queue.cc:260] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.827023 5337 ts_tablet_manager.cc:1398] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.033s user 0.009s sys 0.018s
I20240503 08:13:32.828474 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-5de086cc1f0b497f87b055e745a38e0d received from {username='slave'} at 127.24.123.9:58965
I20240503 08:13:32.828625 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-5de086cc1f0b497f87b055e745a38e0d on tablet 5de086cc1f0b497f87b055e745a38e0d with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:32.831175 5353 tablet_copy_client.cc:523] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:32.835038 5353 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap starting.
I20240503 08:13:32.843451 5317 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:32.843829 5333 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:32.844216 5333 tablet_bootstrap.cc:492] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:32.844482 5333 ts_tablet_manager.cc:1367] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.089s user 0.046s sys 0.012s
I20240503 08:13:32.845105 5320 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:32.845525 5320 tablet_bootstrap.cc:492] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:32.845377 5333 raft_consensus.cc:348] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.845693 5333 raft_consensus.cc:729] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:32.845777 5320 ts_tablet_manager.cc:1367] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.104s user 0.045s sys 0.013s
I20240503 08:13:32.845896 5333 consensus_queue.cc:260] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.846587 5333 ts_tablet_manager.cc:1398] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:32.846931 5317 tablet_bootstrap.cc:492] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:32.847177 5317 ts_tablet_manager.cc:1367] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.118s user 0.053s sys 0.001s
I20240503 08:13:32.847937 5317 raft_consensus.cc:348] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.848174 5317 raft_consensus.cc:729] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:32.848359 5317 consensus_queue.cc:260] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.848981 5317 ts_tablet_manager.cc:1398] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:32.849598 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-734d2de8d2804b2c8d2f144157e12f77 received from {username='slave'} at 127.24.123.9:42783
I20240503 08:13:32.849747 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-734d2de8d2804b2c8d2f144157e12f77 on tablet 734d2de8d2804b2c8d2f144157e12f77 with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:32.853933 5320 raft_consensus.cc:348] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.854166 5320 raft_consensus.cc:729] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:32.854337 5320 consensus_queue.cc:260] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.854946 5320 ts_tablet_manager.cc:1398] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.009s user 0.003s sys 0.001s
I20240503 08:13:32.855782 32552 tablet_copy_service.cc:342] P 09a050703a1a4c0a8dfc04e4c2853797: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-c970023069d44323a2b38629da1e2fad received from {username='slave'} at 127.24.123.9:42783
I20240503 08:13:32.855906 32552 tablet_copy_service.cc:433] P 09a050703a1a4c0a8dfc04e4c2853797: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-c970023069d44323a2b38629da1e2fad on tablet c970023069d44323a2b38629da1e2fad with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:32.856312 32292 tablet_copy_service.cc:342] P 272c59640332489181cbf2cc52d266bb: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-d9151dcb76174b27991fdcf3839db0a5 received from {username='slave'} at 127.24.123.9:58965
I20240503 08:13:32.856443 32292 tablet_copy_service.cc:433] P 272c59640332489181cbf2cc52d266bb: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-d9151dcb76174b27991fdcf3839db0a5 on tablet d9151dcb76174b27991fdcf3839db0a5 with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:32.890569 5353 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:32.890966 5353 tablet_bootstrap.cc:492] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: Bootstrap complete.
I20240503 08:13:32.891203 5353 ts_tablet_manager.cc:1367] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent bootstrapping tablet: real 0.056s user 0.036s sys 0.010s
I20240503 08:13:32.891808 5353 raft_consensus.cc:348] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.892010 5353 raft_consensus.cc:729] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 237c0b0fe44549a89d1b5bfc8c55da91, State: Initialized, Role: LEARNER
I20240503 08:13:32.892194 5353 consensus_queue.cc:260] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } }
I20240503 08:13:32.892700 5353 ts_tablet_manager.cc:1398] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:32.893479 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-7cc8a3241af248578b6ce87b1e33cea9 received from {username='slave'} at 127.24.123.9:60153
I20240503 08:13:32.893631 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session 237c0b0fe44549a89d1b5bfc8c55da91-7cc8a3241af248578b6ce87b1e33cea9 on tablet 7cc8a3241af248578b6ce87b1e33cea9 with peer 237c0b0fe44549a89d1b5bfc8c55da91
I20240503 08:13:32.951687 2841 raft_consensus.cc:1203] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:33.120333 2841 raft_consensus.cc:1203] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:33.243160 2841 raft_consensus.cc:1203] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:33.283916 2841 raft_consensus.cc:1203] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:33.377202 2841 raft_consensus.cc:1203] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:33.474635 5664 raft_consensus.cc:1050] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:33.475621 5664 consensus_queue.cc:237] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:33.478535 1988 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.478742 2841 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.479112 32270 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.479287 5666 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.479666 5664 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.480064 5637 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.483177 5637 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.483774 2841 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.484565 32270 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.489017 1988 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.489156 31992 catalog_manager.cc:5484] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 reported cstate change: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.580150 5012 raft_consensus.cc:1050] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:33.580958 5012 consensus_queue.cc:237] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:33.582070 32140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9"
dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39976
I20240503 08:13:33.582309 32140 raft_consensus.cc:595] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:33.584906 2841 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.585475 5012 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.597424 32140 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.597414 1589 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.598058 5685 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.598596 5012 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.601295 5012 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.604527 1589 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.604802 2841 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.608935 31991 catalog_manager.cc:5484] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.615622 32270 consensus_queue.cc:237] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:33.617580 2841 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:33.617722 1589 raft_consensus.cc:1261] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:33.618448 5688 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:33.618857 5685 consensus_queue.cc:1035] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:33.621392 5684 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.622115 2841 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.622161 1589 raft_consensus.cc:2932] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.625059 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5de086cc1f0b497f87b055e745a38e0d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:33.626814 31995 catalog_manager.cc:5484] T 5de086cc1f0b497f87b055e745a38e0d P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.631279 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de086cc1f0b497f87b055e745a38e0d with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:33.635907 5702 tablet_replica.cc:330] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:33.636171 5702 raft_consensus.cc:2229] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:33.636421 5702 pending_rounds.cc:62] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: Trying to abort 1 pending ops.
I20240503 08:13:33.636543 5702 pending_rounds.cc:68] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023509559625601024 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5de086cc1f0b497f87b055e745a38e0d" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } } }
I20240503 08:13:33.637022 5702 raft_consensus.cc:2894] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 08:13:33.637203 5702 raft_consensus.cc:2866] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:13:33.637368 5702 raft_consensus.cc:2258] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:33.638306 5702 ts_tablet_manager.cc:1875] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:33.644948 5702 ts_tablet_manager.cc:1888] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:33.645098 5702 log.cc:1196] T 5de086cc1f0b497f87b055e745a38e0d P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/5de086cc1f0b497f87b055e745a38e0d
I20240503 08:13:33.645954 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet 5de086cc1f0b497f87b055e745a38e0d (table rebalance_test_table_5 [id=30d2aa2df22148d2b821d3077667ef88]) successfully deleted
I20240503 08:13:33.681191 32140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9"
dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40000
I20240503 08:13:33.681391 32140 raft_consensus.cc:595] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:33.704046 5738 raft_consensus.cc:1050] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:33.704720 5738 consensus_queue.cc:237] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:33.707921 1723 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.708189 2841 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.708191 32140 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.708747 4601 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.709040 5638 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.709556 4601 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.725956 5638 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.727512 32140 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.732394 31991 catalog_manager.cc:5484] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:33.731531 1724 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.735675 2841 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.741791 32532 consensus_queue.cc:237] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:33.743362 1724 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:33.743381 2841 raft_consensus.cc:1261] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:33.743865 5738 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:33.744168 5638 consensus_queue.cc:1035] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:33.746358 4601 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.747133 1724 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.747236 2841 raft_consensus.cc:2932] T c970023069d44323a2b38629da1e2fad P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.750056 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c970023069d44323a2b38629da1e2fad with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:33.751605 31991 catalog_manager.cc:5484] T c970023069d44323a2b38629da1e2fad P 6c93e47973e047d0a162a50f72f9a757 reported cstate change: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.755462 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet c970023069d44323a2b38629da1e2fad with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:33.755846 5702 tablet_replica.cc:330] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:33.756037 5702 raft_consensus.cc:2229] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:33.756319 5702 raft_consensus.cc:2258] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:33.757226 5702 ts_tablet_manager.cc:1875] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:33.762156 5702 ts_tablet_manager.cc:1888] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:33.762249 5702 log.cc:1196] T c970023069d44323a2b38629da1e2fad P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/c970023069d44323a2b38629da1e2fad
I20240503 08:13:33.762988 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet c970023069d44323a2b38629da1e2fad (table rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]) successfully deleted
I20240503 08:13:33.859869 5743 raft_consensus.cc:1050] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:33.860718 5743 consensus_queue.cc:237] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:33.862993 1724 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.863117 32140 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.863457 2841 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.863639 5638 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.863929 5738 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.864158 5743 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.866720 5738 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.867233 2841 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.872224 32000 catalog_manager.cc:5484] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91 reported cstate change: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.873365 32140 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.882011 1724 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.884330 32532 consensus_queue.cc:237] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:33.886138 1724 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:33.886396 2841 raft_consensus.cc:1261] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Refusing update from remote peer 09a050703a1a4c0a8dfc04e4c2853797: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:33.886597 4601 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:33.886914 5738 consensus_queue.cc:1035] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:33.888809 5638 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.889411 1724 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 6c93e47973e047d0a162a50f72f9a757 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.891088 2841 raft_consensus.cc:2932] T 734d2de8d2804b2c8d2f144157e12f77 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.892329 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 734d2de8d2804b2c8d2f144157e12f77 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:33.893996 31995 catalog_manager.cc:5484] T 734d2de8d2804b2c8d2f144157e12f77 P 09a050703a1a4c0a8dfc04e4c2853797 reported cstate change: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "09a050703a1a4c0a8dfc04e4c2853797" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:33.898046 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet 734d2de8d2804b2c8d2f144157e12f77 with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:33.898406 5702 tablet_replica.cc:330] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:33.898649 5702 raft_consensus.cc:2229] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:33.899121 5702 raft_consensus.cc:2258] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:33.899948 5702 ts_tablet_manager.cc:1875] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:33.904222 5702 ts_tablet_manager.cc:1888] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:33.904321 5702 log.cc:1196] T 734d2de8d2804b2c8d2f144157e12f77 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/734d2de8d2804b2c8d2f144157e12f77
I20240503 08:13:33.904902 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet 734d2de8d2804b2c8d2f144157e12f77 (table rebalance_test_table_2 [id=0c9053a015e2409a9d2bb995dc93a0eb]) successfully deleted
I20240503 08:13:33.923949 32140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9"
dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40016
I20240503 08:13:33.924147 32140 raft_consensus.cc:595] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:33.977802 5685 raft_consensus.cc:1050] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb: attempting to promote NON_VOTER 237c0b0fe44549a89d1b5bfc8c55da91 to VOTER
I20240503 08:13:33.978662 5685 consensus_queue.cc:237] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:33.980068 32140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9"
dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40030
I20240503 08:13:33.980281 32140 raft_consensus.cc:595] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:33.981324 32140 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.981537 1589 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.981555 2841 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 LEARNER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:33.982035 5688 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.982422 5684 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.983623 5687 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:33.987664 5688 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.988605 32141 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.991396 1589 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.991451 2841 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:33.992422 32000 catalog_manager.cc:5484] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 7 to 8, 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:33.997823 32270 consensus_queue.cc:237] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:33.999722 2841 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:33.999752 1589 raft_consensus.cc:1261] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:34.000262 5684 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:34.000648 5687 consensus_queue.cc:1035] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:34.002331 5688 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 272c59640332489181cbf2cc52d266bb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:34.003121 2844 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:34.003211 1589 raft_consensus.cc:2932] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:34.005811 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9151dcb76174b27991fdcf3839db0a5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:34.007617 31991 catalog_manager.cc:5484] T d9151dcb76174b27991fdcf3839db0a5 P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 8 to 9, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:34.011072 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet d9151dcb76174b27991fdcf3839db0a5 with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:34.011442 5702 tablet_replica.cc:330] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:34.011631 5702 raft_consensus.cc:2229] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:34.011956 5702 raft_consensus.cc:2258] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:34.012851 5702 ts_tablet_manager.cc:1875] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:34.017701 5702 ts_tablet_manager.cc:1888] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:34.017796 5702 log.cc:1196] T d9151dcb76174b27991fdcf3839db0a5 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/d9151dcb76174b27991fdcf3839db0a5
I20240503 08:13:34.018360 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet d9151dcb76174b27991fdcf3839db0a5 (table rebalance_test_table_8 [id=e34ce68d10484c75b2f1715a89ef76a5]) successfully deleted
I20240503 08:13:34.025604 5040 raft_consensus.cc:979] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: : Instructing follower 272c59640332489181cbf2cc52d266bb to start an election
I20240503 08:13:34.025736 5040 raft_consensus.cc:1067] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Signalling peer 272c59640332489181cbf2cc52d266bb to start an election
I20240503 08:13:34.026492 32270 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9"
dest_uuid: "272c59640332489181cbf2cc52d266bb"
from {username='slave'} at 127.24.123.1:47541
I20240503 08:13:34.026713 32270 raft_consensus.cc:482] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:34.026839 32270 raft_consensus.cc:3037] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:34.029078 32270 raft_consensus.cc:504] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:34.030149 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:34.030320 32141 raft_consensus.cc:3032] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:34.030426 32141 raft_consensus.cc:729] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:34.030683 32270 leader_election.cc:288] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 2 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749), 237c0b0fe44549a89d1b5bfc8c55da91 (127.24.123.9:41169)
I20240503 08:13:34.030722 32141 consensus_queue.cc:260] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:34.030903 2844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "237c0b0fe44549a89d1b5bfc8c55da91"
I20240503 08:13:34.031181 32141 raft_consensus.cc:3037] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:34.031247 2844 raft_consensus.cc:3037] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:34.032944 32141 raft_consensus.cc:2454] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 2.
I20240503 08:13:34.033334 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9" candidate_uuid: "272c59640332489181cbf2cc52d266bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e615bd1588964e2fbe14f14f4aae0a98"
I20240503 08:13:34.033526 1988 raft_consensus.cc:3037] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:34.034477 2844 raft_consensus.cc:2454] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 2.
I20240503 08:13:34.035617 32205 leader_election.cc:302] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 237c0b0fe44549a89d1b5bfc8c55da91, 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90; no voters:
I20240503 08:13:34.036049 5687 raft_consensus.cc:2781] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:34.036185 5687 raft_consensus.cc:686] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 2 LEADER]: Becoming Leader. State: Replica: 272c59640332489181cbf2cc52d266bb, State: Running, Role: LEADER
I20240503 08:13:34.036470 1988 raft_consensus.cc:2454] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 272c59640332489181cbf2cc52d266bb in term 2.
I20240503 08:13:34.036509 5687 consensus_queue.cc:237] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:34.039503 31995 catalog_manager.cc:5484] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb reported cstate change: term changed from 1 to 2, leader changed from 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) to 272c59640332489181cbf2cc52d266bb (127.24.123.2). New cstate: current_term: 2 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:34.042322 32141 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cc8a3241af248578b6ce87b1e33cea9"
dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40054
I20240503 08:13:34.042477 32141 raft_consensus.cc:595] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 08:13:34.042572 32141 raft_consensus.cc:603] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 08:13:34.422824 2844 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:34.423410 5684 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:34.493245 32141 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:34.493929 5687 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:34.497757 1988 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:34.498680 5685 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:34.921597 32270 consensus_queue.cc:237] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:34.923195 2844 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:34.923203 1988 raft_consensus.cc:1261] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Refusing update from remote peer 272c59640332489181cbf2cc52d266bb: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:34.923691 5685 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:34.924003 5687 consensus_queue.cc:1035] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:34.926015 5687 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:34.926731 2844 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:34.928859 31985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7cc8a3241af248578b6ce87b1e33cea9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:34.929360 1988 raft_consensus.cc:2932] T 7cc8a3241af248578b6ce87b1e33cea9 P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } } }
I20240503 08:13:34.930297 31995 catalog_manager.cc:5484] T 7cc8a3241af248578b6ce87b1e33cea9 P 272c59640332489181cbf2cc52d266bb reported cstate change: config changed from index 8 to 10, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 2 leader_uuid: "272c59640332489181cbf2cc52d266bb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:34.935496 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cc8a3241af248578b6ce87b1e33cea9 with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:34.936197 5835 tablet_replica.cc:330] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:34.936370 5835 raft_consensus.cc:2229] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:34.936537 5835 raft_consensus.cc:2258] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:34.937338 5835 ts_tablet_manager.cc:1875] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:34.941587 5835 ts_tablet_manager.cc:1888] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:34.941701 5835 log.cc:1196] T 7cc8a3241af248578b6ce87b1e33cea9 P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/7cc8a3241af248578b6ce87b1e33cea9
I20240503 08:13:34.942409 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet 7cc8a3241af248578b6ce87b1e33cea9 (table rebalance_test_table_6 [id=afd5786d24e5470bb21c4f837013e719]) successfully deleted
I20240503 08:13:35.259119 32380 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:35.272940 1837 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:35.275259 32119 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:35.276413 1969 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:35.278564 1571 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:35.285200 32251 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:35.285323 2825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:35.304098 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:35.306509 1705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:35.413393 1589 consensus_queue.cc:237] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:35.417266 32270 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:13:35.417297 32400 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:13:35.417910 5894 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:35.418305 5358 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:35.420984 5358 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:35.421550 32270 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
W20240503 08:13:35.425909 1525 consensus_peers.cc:485] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a -> Peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Couldn't send request to peer e615bd1588964e2fbe14f14f4aae0a98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84bef4a86df446d0b4ff490822068fbb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:35.427493 31999 catalog_manager.cc:5484] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 7 to 8, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New cstate: current_term: 2 leader_uuid: "c6b280e8ecd54855b7b21b58a223149a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:35.441280 32400 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } } }
I20240503 08:13:35.462508 32141 consensus_queue.cc:237] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:35.465171 1854 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:35.465209 32074 consensus_peers.cc:485] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86b7e012f1544986a9460cd5dac5cece. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:35.466003 5040 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:35.467985 32400 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:35.468946 5907 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:35.470324 5905 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:35.471980 1854 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:35.471997 32400 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } } }
I20240503 08:13:35.476716 31992 catalog_manager.cc:5484] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 reported cstate change: config changed from index 6 to 7, NON_VOTER c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) added. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:35.805491 5930 ts_tablet_manager.cc:902] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: Initiating tablet copy from peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035)
I20240503 08:13:35.806963 5930 tablet_copy_client.cc:308] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:44035
I20240503 08:13:35.813377 1611 tablet_copy_service.cc:140] P c6b280e8ecd54855b7b21b58a223149a: Received BeginTabletCopySession request for tablet 84bef4a86df446d0b4ff490822068fbb from peer e615bd1588964e2fbe14f14f4aae0a98 ({username='slave'} at 127.24.123.8:58955)
I20240503 08:13:35.813593 1611 tablet_copy_service.cc:161] P c6b280e8ecd54855b7b21b58a223149a: Beginning new tablet copy session on tablet 84bef4a86df446d0b4ff490822068fbb from peer e615bd1588964e2fbe14f14f4aae0a98 at {username='slave'} at 127.24.123.8:58955: session id = e615bd1588964e2fbe14f14f4aae0a98-84bef4a86df446d0b4ff490822068fbb
I20240503 08:13:35.816148 1611 tablet_copy_source_session.cc:205] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:35.817512 5930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bef4a86df446d0b4ff490822068fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.822023 5930 tablet_copy_client.cc:783] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:35.822239 5930 tablet_copy_client.cc:647] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:35.823724 5934 tablet_copy_client.cc:1002] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:35.862092 5936 ts_tablet_manager.cc:902] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: Initiating tablet copy from peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
I20240503 08:13:35.863492 5936 tablet_copy_client.cc:308] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:41731
I20240503 08:13:35.864183 32161 tablet_copy_service.cc:140] P 99e2f0f92de9442c9e73d00d03ca6b90: Received BeginTabletCopySession request for tablet 86b7e012f1544986a9460cd5dac5cece from peer c6b280e8ecd54855b7b21b58a223149a ({username='slave'} at 127.24.123.5:56869)
I20240503 08:13:35.864351 32161 tablet_copy_service.cc:161] P 99e2f0f92de9442c9e73d00d03ca6b90: Beginning new tablet copy session on tablet 86b7e012f1544986a9460cd5dac5cece from peer c6b280e8ecd54855b7b21b58a223149a at {username='slave'} at 127.24.123.5:56869: session id = c6b280e8ecd54855b7b21b58a223149a-86b7e012f1544986a9460cd5dac5cece
I20240503 08:13:35.866225 32161 tablet_copy_source_session.cc:205] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:35.867391 5936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b7e012f1544986a9460cd5dac5cece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.871807 5936 tablet_copy_client.cc:783] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:35.872022 5936 tablet_copy_client.cc:647] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:35.873482 5940 tablet_copy_client.cc:1002] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Injecting latency into file download: 1500
I20240503 08:13:37.324079 5930 tablet_copy_client.cc:523] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:37.326623 5930 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap starting.
I20240503 08:13:37.357501 5930 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:37.357870 5930 tablet_bootstrap.cc:492] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: Bootstrap complete.
I20240503 08:13:37.358140 5930 ts_tablet_manager.cc:1367] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.004s
I20240503 08:13:37.358829 5930 raft_consensus.cc:348] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:37.359095 5930 raft_consensus.cc:729] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e615bd1588964e2fbe14f14f4aae0a98, State: Initialized, Role: LEARNER
I20240503 08:13:37.359279 5930 consensus_queue.cc:260] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: true } }
I20240503 08:13:37.359782 5930 ts_tablet_manager.cc:1398] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:37.360683 1611 tablet_copy_service.cc:342] P c6b280e8ecd54855b7b21b58a223149a: Request end of tablet copy session e615bd1588964e2fbe14f14f4aae0a98-84bef4a86df446d0b4ff490822068fbb received from {username='slave'} at 127.24.123.8:58955
I20240503 08:13:37.360858 1611 tablet_copy_service.cc:433] P c6b280e8ecd54855b7b21b58a223149a: ending tablet copy session e615bd1588964e2fbe14f14f4aae0a98-84bef4a86df446d0b4ff490822068fbb on tablet 84bef4a86df446d0b4ff490822068fbb with peer e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:37.373826 5936 tablet_copy_client.cc:523] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:37.376436 5936 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: Bootstrap starting.
I20240503 08:13:37.407599 5936 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:37.407976 5936 tablet_bootstrap.cc:492] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: Bootstrap complete.
I20240503 08:13:37.408208 5936 ts_tablet_manager.cc:1367] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: Time spent bootstrapping tablet: real 0.032s user 0.030s sys 0.001s
I20240503 08:13:37.408898 5936 raft_consensus.cc:348] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:37.409119 5936 raft_consensus.cc:729] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b280e8ecd54855b7b21b58a223149a, State: Initialized, Role: LEARNER
I20240503 08:13:37.409312 5936 consensus_queue.cc:260] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: NON_VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: true } }
I20240503 08:13:37.409812 5936 ts_tablet_manager.cc:1398] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:37.410573 32161 tablet_copy_service.cc:342] P 99e2f0f92de9442c9e73d00d03ca6b90: Request end of tablet copy session c6b280e8ecd54855b7b21b58a223149a-86b7e012f1544986a9460cd5dac5cece received from {username='slave'} at 127.24.123.5:56869
I20240503 08:13:37.410708 32161 tablet_copy_service.cc:433] P 99e2f0f92de9442c9e73d00d03ca6b90: ending tablet copy session c6b280e8ecd54855b7b21b58a223149a-86b7e012f1544986a9460cd5dac5cece on tablet 86b7e012f1544986a9460cd5dac5cece with peer c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:37.635396 1988 raft_consensus.cc:1203] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:13:37.805658 1591 raft_consensus.cc:1203] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:38.072875 6052 raft_consensus.cc:1050] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a: attempting to promote NON_VOTER e615bd1588964e2fbe14f14f4aae0a98 to VOTER
I20240503 08:13:38.073624 6052 consensus_queue.cc:237] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:38.076628 32270 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:38.076651 1988 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98 [term 2 LEARNER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:38.076809 32400 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:38.077193 6052 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:38.077553 6055 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:38.077937 6053 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:38.080634 6053 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:38.081446 32400 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:38.081588 1988 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:38.085135 32270 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:38.090695 31994 catalog_manager.cc:5484] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 8 to 9, e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c6b280e8ecd54855b7b21b58a223149a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:38.110641 1591 consensus_queue.cc:237] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:38.112473 1988 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:38.113487 6055 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:38.115291 32400 raft_consensus.cc:1261] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer c6b280e8ecd54855b7b21b58a223149a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:38.116776 6055 consensus_queue.cc:1035] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:38.117540 6055 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:38.121030 1988 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:38.123498 32400 raft_consensus.cc:2932] T 84bef4a86df446d0b4ff490822068fbb P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:38.128291 31984 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84bef4a86df446d0b4ff490822068fbb with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:38.128118 31991 catalog_manager.cc:5484] T 84bef4a86df446d0b4ff490822068fbb P e615bd1588964e2fbe14f14f4aae0a98 reported cstate change: config changed from index 9 to 10, VOTER 272c59640332489181cbf2cc52d266bb (127.24.123.2) evicted. New cstate: current_term: 2 leader_uuid: "c6b280e8ecd54855b7b21b58a223149a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } }
I20240503 08:13:38.132517 32251 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bef4a86df446d0b4ff490822068fbb with delete_type TABLET_DATA_TOMBSTONED (TS 272c59640332489181cbf2cc52d266bb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48250
I20240503 08:13:38.139163 6073 tablet_replica.cc:330] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb: stopping tablet replica
I20240503 08:13:38.139395 6073 raft_consensus.cc:2229] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:38.139796 6073 raft_consensus.cc:2258] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:38.140676 6073 ts_tablet_manager.cc:1875] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:38.146238 6073 ts_tablet_manager.cc:1888] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:38.146358 6073 log.cc:1196] T 84bef4a86df446d0b4ff490822068fbb P 272c59640332489181cbf2cc52d266bb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/84bef4a86df446d0b4ff490822068fbb
I20240503 08:13:38.147151 31985 catalog_manager.cc:4830] TS 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): tablet 84bef4a86df446d0b4ff490822068fbb (table rebalance_test_table_7 [id=038d7040373f4817883d0b4461e6e277]) successfully deleted
I20240503 08:13:38.412484 5905 raft_consensus.cc:1050] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: attempting to promote NON_VOTER c6b280e8ecd54855b7b21b58a223149a to VOTER
I20240503 08:13:38.413143 5905 consensus_queue.cc:237] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:38.415565 1854 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:38.415750 32400 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:38.416131 5905 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:38.416342 1591 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 1 LEARNER]: Refusing update from remote peer 99e2f0f92de9442c9e73d00d03ca6b90: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:38.416488 6087 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:38.417018 5905 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:38.420152 5905 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:38.423290 1591 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:38.427747 1854 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:38.428875 31992 catalog_manager.cc:5484] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a reported cstate change: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:38.433885 32400 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6b280e8ecd54855b7b21b58a223149a (127.24.123.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:38.579648 32141 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece"
dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47462
I20240503 08:13:38.579838 32141 raft_consensus.cc:595] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:38.796456 32141 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece"
dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47478
I20240503 08:13:38.796643 32141 raft_consensus.cc:595] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:38.944109 6091 raft_consensus.cc:979] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: : Instructing follower f7f6592d13d245508261964ca6ffc5c4 to start an election
I20240503 08:13:38.944286 6091 raft_consensus.cc:1067] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Signalling peer f7f6592d13d245508261964ca6ffc5c4 to start an election
I20240503 08:13:38.944893 1854 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece"
dest_uuid: "f7f6592d13d245508261964ca6ffc5c4"
from {username='slave'} at 127.24.123.1:60753
I20240503 08:13:38.945111 1854 raft_consensus.cc:482] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:38.945236 1854 raft_consensus.cc:3037] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:38.947345 1854 raft_consensus.cc:504] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:38.948249 1854 leader_election.cc:288] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [CANDIDATE]: Term 2 election: Requested vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035)
I20240503 08:13:38.948464 32141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece" candidate_uuid: "f7f6592d13d245508261964ca6ffc5c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "99e2f0f92de9442c9e73d00d03ca6b90"
I20240503 08:13:38.948689 32141 raft_consensus.cc:3032] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:38.948776 32141 raft_consensus.cc:729] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99e2f0f92de9442c9e73d00d03ca6b90, State: Running, Role: LEADER
I20240503 08:13:38.949050 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece" candidate_uuid: "f7f6592d13d245508261964ca6ffc5c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "71ad6bf07682497aaa1e9429320aef79"
I20240503 08:13:38.949070 32141 consensus_queue.cc:260] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:38.949262 32400 raft_consensus.cc:3037] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:38.949515 32141 raft_consensus.cc:3037] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:38.951601 32400 raft_consensus.cc:2454] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f6592d13d245508261964ca6ffc5c4 in term 2.
I20240503 08:13:38.952312 32141 raft_consensus.cc:2454] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f6592d13d245508261964ca6ffc5c4 in term 2.
I20240503 08:13:38.952783 1789 leader_election.cc:302] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90, f7f6592d13d245508261964ca6ffc5c4; no voters:
I20240503 08:13:38.953069 5307 raft_consensus.cc:2781] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:38.953399 5307 raft_consensus.cc:686] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 2 LEADER]: Becoming Leader. State: Replica: f7f6592d13d245508261964ca6ffc5c4, State: Running, Role: LEADER
I20240503 08:13:38.953727 5307 consensus_queue.cc:237] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:38.956830 31999 catalog_manager.cc:5484] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 reported cstate change: term changed from 1 to 2, leader changed from 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) to f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7). New cstate: current_term: 2 leader_uuid: "f7f6592d13d245508261964ca6ffc5c4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:38.999075 1591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b7e012f1544986a9460cd5dac5cece" candidate_uuid: "f7f6592d13d245508261964ca6ffc5c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c6b280e8ecd54855b7b21b58a223149a"
I20240503 08:13:38.999387 1591 raft_consensus.cc:3037] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:39.002045 1591 raft_consensus.cc:2454] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f6592d13d245508261964ca6ffc5c4 in term 2.
I20240503 08:13:39.337580 1591 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:39.338129 5307 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:39.381894 32400 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:39.382386 5307 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:39.387192 32141 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:39.387928 5307 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:39.390365 1854 consensus_queue.cc:237] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:39.392462 32400 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:39.392571 1591 raft_consensus.cc:1261] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 2 FOLLOWER]: Refusing update from remote peer f7f6592d13d245508261964ca6ffc5c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:39.393278 6114 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:39.393682 6131 consensus_queue.cc:1035] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:39.395646 6114 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:39.396960 1591 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P c6b280e8ecd54855b7b21b58a223149a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:39.398439 31986 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86b7e012f1544986a9460cd5dac5cece with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:39.398550 32400 raft_consensus.cc:2932] T 86b7e012f1544986a9460cd5dac5cece P 71ad6bf07682497aaa1e9429320aef79 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } }
I20240503 08:13:39.400099 31999 catalog_manager.cc:5484] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 reported cstate change: config changed from index 8 to 10, VOTER 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1) evicted. New cstate: current_term: 2 leader_uuid: "f7f6592d13d245508261964ca6ffc5c4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:39.403667 32119 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b7e012f1544986a9460cd5dac5cece with delete_type TABLET_DATA_TOMBSTONED (TS 99e2f0f92de9442c9e73d00d03ca6b90 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43600
I20240503 08:13:39.407315 6135 tablet_replica.cc:330] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: stopping tablet replica
I20240503 08:13:39.407508 6135 raft_consensus.cc:2229] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:39.407725 6135 raft_consensus.cc:2258] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:39.408469 6135 ts_tablet_manager.cc:1875] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:39.412819 6135 ts_tablet_manager.cc:1888] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:39.412921 6135 log.cc:1196] T 86b7e012f1544986a9460cd5dac5cece P 99e2f0f92de9442c9e73d00d03ca6b90: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/86b7e012f1544986a9460cd5dac5cece
I20240503 08:13:39.413560 31984 catalog_manager.cc:4830] TS 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): tablet 86b7e012f1544986a9460cd5dac5cece (table rebalance_test_table_1 [id=d6bd54eda0b54123bd0fc5a2df2246aa]) successfully deleted
I20240503 08:13:39.555541 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:39.557893 2825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:39.571147 32251 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:39.572556 32380 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:39.617314 1969 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:39.626744 1837 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:39.641758 32119 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:39.662608 1705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:39.676333 1571 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:39.873281 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:39.873826 2825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:39.885939 32251 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:39.897179 1705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:39.901412 32380 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:39.948966 32119 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:39.975615 1571 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:39.988790 1837 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:39.998700 1969 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:40.863374 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:40.864257 32251 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:40.864285 2825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:40.875293 32119 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:40.880592 1705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:40.900313 1571 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:40.900882 1969 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:40.903033 32380 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:40.939317 1837 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:41.207948 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:41.211066 2825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:41.232096 1705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:41.244557 32251 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:41.274745 32380 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:41.287726 1571 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:41.303059 1969 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:41.303587 32119 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:41.329411 1837 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:41.589569 2825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:41.591193 32251 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:41.593555 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:41.597095 32119 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:41.602831 32380 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:41.608187 1969 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:41.615882 1705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:41.621235 1571 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:41.643220 1837 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:42.036574 32510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:42.036967 2825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:42.044085 1969 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:42.054066 32380 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:13:42.054153 1571 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:42.063169 32119 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:13:42.065204 1837 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:42.067898 1705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:13:42.114300 32251 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
c96f604a3b3640f2b9f75fc4380c08d4 | 127.24.123.62:44195 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.123.20:34105 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
09a050703a1a4c0a8dfc04e4c2853797 | 127.24.123.4:46785 | HEALTHY | <none> | 6 | 0
237c0b0fe44549a89d1b5bfc8c55da91 | 127.24.123.9:41169 | HEALTHY | <none> | 0 | 0
272c59640332489181cbf2cc52d266bb | 127.24.123.2:39617 | HEALTHY | <none> | 8 | 0
6c93e47973e047d0a162a50f72f9a757 | 127.24.123.6:36447 | HEALTHY | <none> | 2 | 0
71ad6bf07682497aaa1e9429320aef79 | 127.24.123.3:34237 | HEALTHY | <none> | 3 | 0
99e2f0f92de9442c9e73d00d03ca6b90 | 127.24.123.1:41731 | HEALTHY | <none> | 7 | 0
c6b280e8ecd54855b7b21b58a223149a | 127.24.123.5:44035 | HEALTHY | <none> | 1 | 0
e615bd1588964e2fbe14f14f4aae0a98 | 127.24.123.8:43749 | HEALTHY | <none> | 1 | 0
f7f6592d13d245508261964ca6ffc5c4 | 127.24.123.7:33603 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 9
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.24.123.1 | experimental | 127.24.123.1:41731
local_ip_for_outbound_sockets | 127.24.123.2 | experimental | 127.24.123.2:39617
local_ip_for_outbound_sockets | 127.24.123.3 | experimental | 127.24.123.3:34237
local_ip_for_outbound_sockets | 127.24.123.4 | experimental | 127.24.123.4:46785
local_ip_for_outbound_sockets | 127.24.123.5 | experimental | 127.24.123.5:44035
local_ip_for_outbound_sockets | 127.24.123.6 | experimental | 127.24.123.6:36447
local_ip_for_outbound_sockets | 127.24.123.7 | experimental | 127.24.123.7:33603
local_ip_for_outbound_sockets | 127.24.123.8 | experimental | 127.24.123.8:43749
local_ip_for_outbound_sockets | 127.24.123.9 | experimental | 127.24.123.9:41169
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.123.1:41731
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.123.2:39617
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.123.3:34237
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.24.123.4:46785
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.24.123.5:44035
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.24.123.6:36447
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.24.123.7:33603
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.24.123.8:43749
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.24.123.9:41169
tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.123.20:34105 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 10
Tablets | 30
Replicas | 90
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:13:42.237447 25068 log_verifier.cc:126] Checking tablet 10646e4785b047d8a3ac6af5ddb06474
I20240503 08:13:42.274634 25068 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 10646e4785b047d8a3ac6af5ddb06474
I20240503 08:13:42.274760 25068 log_verifier.cc:126] Checking tablet 130b43c1e784400fb8851a71428014a1
I20240503 08:13:42.311756 25068 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 130b43c1e784400fb8851a71428014a1
I20240503 08:13:42.311872 25068 log_verifier.cc:126] Checking tablet 25f1956062e84b92a3b036fc5dff4b46
I20240503 08:13:42.349498 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 25f1956062e84b92a3b036fc5dff4b46
I20240503 08:13:42.349602 25068 log_verifier.cc:126] Checking tablet 27a469ffc17447ed86860e991ff3534a
I20240503 08:13:42.384159 25068 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 27a469ffc17447ed86860e991ff3534a
I20240503 08:13:42.384327 25068 log_verifier.cc:126] Checking tablet 2bd736780ea840ab9b6630a4acd80ae9
I20240503 08:13:42.425069 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 2bd736780ea840ab9b6630a4acd80ae9
I20240503 08:13:42.425202 25068 log_verifier.cc:126] Checking tablet 2cf32f9e1e914653a6594d58c67cd80e
I20240503 08:13:42.495357 25068 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2cf32f9e1e914653a6594d58c67cd80e
I20240503 08:13:42.495446 25068 log_verifier.cc:126] Checking tablet 2f4ac5dea8024445ad2cd1ca724f2290
I20240503 08:13:42.531424 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 2f4ac5dea8024445ad2cd1ca724f2290
I20240503 08:13:42.531522 25068 log_verifier.cc:126] Checking tablet 400a0317f9184c99a449f7f160fa998f
I20240503 08:13:42.577616 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 400a0317f9184c99a449f7f160fa998f
I20240503 08:13:42.577723 25068 log_verifier.cc:126] Checking tablet 59a99f630f59472ca5c6778525e79fbe
I20240503 08:13:42.612650 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 59a99f630f59472ca5c6778525e79fbe
I20240503 08:13:42.612746 25068 log_verifier.cc:126] Checking tablet 5de086cc1f0b497f87b055e745a38e0d
I20240503 08:13:42.655154 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 5de086cc1f0b497f87b055e745a38e0d
I20240503 08:13:42.655360 25068 log_verifier.cc:126] Checking tablet 734d2de8d2804b2c8d2f144157e12f77
I20240503 08:13:42.697296 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 734d2de8d2804b2c8d2f144157e12f77
I20240503 08:13:42.697434 25068 log_verifier.cc:126] Checking tablet 7cc8a3241af248578b6ce87b1e33cea9
I20240503 08:13:42.769634 25068 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 7cc8a3241af248578b6ce87b1e33cea9
I20240503 08:13:42.769726 25068 log_verifier.cc:126] Checking tablet 7d3ee53044cd43debe6b59a2ef51c230
I20240503 08:13:42.804998 25068 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 7d3ee53044cd43debe6b59a2ef51c230
I20240503 08:13:42.805092 25068 log_verifier.cc:126] Checking tablet 84bef4a86df446d0b4ff490822068fbb
I20240503 08:13:42.841663 25068 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 84bef4a86df446d0b4ff490822068fbb
I20240503 08:13:42.841791 25068 log_verifier.cc:126] Checking tablet 86b7e012f1544986a9460cd5dac5cece
I20240503 08:13:42.879505 25068 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 86b7e012f1544986a9460cd5dac5cece
I20240503 08:13:42.879930 25068 log_verifier.cc:126] Checking tablet 8e6a81a3850e49edb08c8060a45b6831
I20240503 08:13:42.916712 25068 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 8e6a81a3850e49edb08c8060a45b6831
I20240503 08:13:42.916836 25068 log_verifier.cc:126] Checking tablet 95491576997c4392b482b4627760d818
I20240503 08:13:42.956521 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 95491576997c4392b482b4627760d818
I20240503 08:13:42.956645 25068 log_verifier.cc:126] Checking tablet a4c5c3fc4dc146d4818d6fdfbd4ac695
I20240503 08:13:43.030485 25068 log_verifier.cc:177] Verified matching terms for 10 ops in tablet a4c5c3fc4dc146d4818d6fdfbd4ac695
I20240503 08:13:43.030575 25068 log_verifier.cc:126] Checking tablet af5c3865ca694bf090dce19f1100e5eb
I20240503 08:13:43.065644 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet af5c3865ca694bf090dce19f1100e5eb
I20240503 08:13:43.065757 25068 log_verifier.cc:126] Checking tablet afdc26bd51bd467f88d7892edb81ca9a
I20240503 08:13:43.100399 25068 log_verifier.cc:177] Verified matching terms for 6 ops in tablet afdc26bd51bd467f88d7892edb81ca9a
I20240503 08:13:43.100490 25068 log_verifier.cc:126] Checking tablet b44429c4434d47c090f9857389c70702
I20240503 08:13:43.136026 25068 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b44429c4434d47c090f9857389c70702
I20240503 08:13:43.136118 25068 log_verifier.cc:126] Checking tablet b7996d398e3440ebbb8f942bda8f6833
I20240503 08:13:43.171636 25068 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b7996d398e3440ebbb8f942bda8f6833
I20240503 08:13:43.171734 25068 log_verifier.cc:126] Checking tablet b7ed99e3dd124de09353f6ed509f3eb1
I20240503 08:13:43.214730 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet b7ed99e3dd124de09353f6ed509f3eb1
I20240503 08:13:43.214854 25068 log_verifier.cc:126] Checking tablet c970023069d44323a2b38629da1e2fad
I20240503 08:13:43.281989 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet c970023069d44323a2b38629da1e2fad
I20240503 08:13:43.282097 25068 log_verifier.cc:126] Checking tablet d9151dcb76174b27991fdcf3839db0a5
I20240503 08:13:43.321372 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet d9151dcb76174b27991fdcf3839db0a5
I20240503 08:13:43.321493 25068 log_verifier.cc:126] Checking tablet e4e25ba167a248ce883fbafc4f6f5fd6
I20240503 08:13:43.358002 25068 log_verifier.cc:177] Verified matching terms for 10 ops in tablet e4e25ba167a248ce883fbafc4f6f5fd6
I20240503 08:13:43.358131 25068 log_verifier.cc:126] Checking tablet f4bd617fecd1483e944365252edf852c
I20240503 08:13:43.394510 25068 log_verifier.cc:177] Verified matching terms for 10 ops in tablet f4bd617fecd1483e944365252edf852c
I20240503 08:13:43.394608 25068 log_verifier.cc:126] Checking tablet f6b5cbb6708049e6ae32da12bb27368f
I20240503 08:13:43.434741 25068 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f6b5cbb6708049e6ae32da12bb27368f
I20240503 08:13:43.434852 25068 log_verifier.cc:126] Checking tablet f9541e40695844a1b05ca1bc0dd8a10d
I20240503 08:13:43.470746 25068 log_verifier.cc:177] Verified matching terms for 9 ops in tablet f9541e40695844a1b05ca1bc0dd8a10d
I20240503 08:13:43.470853 25068 log_verifier.cc:126] Checking tablet feee2677e29f4614825042f6fb663389
I20240503 08:13:43.530150 25068 log_verifier.cc:177] Verified matching terms for 7 ops in tablet feee2677e29f4614825042f6fb663389
I20240503 08:13:43.531194 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 32059
I20240503 08:13:43.537801 32181 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:43.771401 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 32059
W20240503 08:13:43.802520 1921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.123.1:41731 (error 108)
W20240503 08:13:43.802736 32208 connection.cc:535] server connection from 127.24.123.1:47541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:43.803735 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 32190
W20240503 08:13:43.805140 1921 consensus_peers.cc:485] T f4bd617fecd1483e944365252edf852c P e615bd1588964e2fbe14f14f4aae0a98 -> Peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): Couldn't send request to peer 99e2f0f92de9442c9e73d00d03ca6b90. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:41731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:43.809273 32312 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:43.814896 1789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.1:41731: connect: Connection refused (error 111)
W20240503 08:13:43.816144 1789 consensus_peers.cc:485] T a4c5c3fc4dc146d4818d6fdfbd4ac695 P f7f6592d13d245508261964ca6ffc5c4 -> Peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): Couldn't send request to peer 99e2f0f92de9442c9e73d00d03ca6b90. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:41731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:44.033495 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 32190
W20240503 08:13:44.065248 1525 connection.cc:535] server connection from 127.24.123.2:37503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:44.065389 1790 connection.cc:535] server connection from 127.24.123.2:48965 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:44.065837 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 32320
W20240503 08:13:44.066221 2775 connection.cc:535] server connection from 127.24.123.2:35401 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:44.071272 32440 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:44.198302 1658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.2:39617: connect: Connection refused (error 111)
W20240503 08:13:44.199785 1658 consensus_peers.cc:485] T e4e25ba167a248ce883fbafc4f6f5fd6 P 6c93e47973e047d0a162a50f72f9a757 -> Peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): Couldn't send request to peer 272c59640332489181cbf2cc52d266bb. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.2:39617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:44.279295 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 32320
I20240503 08:13:44.308614 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 32450
W20240503 08:13:44.309681 1789 consensus_peers.cc:485] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 -> Peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): Couldn't send request to peer 71ad6bf07682497aaa1e9429320aef79. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.3:34237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:44.314149 32571 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:44.322554 1523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.3:34237: connect: Connection refused (error 111)
W20240503 08:13:44.323994 1523 consensus_peers.cc:485] T 84bef4a86df446d0b4ff490822068fbb P c6b280e8ecd54855b7b21b58a223149a -> Peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): Couldn't send request to peer 71ad6bf07682497aaa1e9429320aef79. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.3:34237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:44.518986 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 32450
W20240503 08:13:44.549548 2772 connection.cc:535] server connection from 127.24.123.4:52035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:44.549574 1660 connection.cc:535] server connection from 127.24.123.4:59771 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:44.550300 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 1508
W20240503 08:13:44.550356 1658 consensus_peers.cc:485] T 8e6a81a3850e49edb08c8060a45b6831 P 6c93e47973e047d0a162a50f72f9a757 -> Peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): Couldn't send request to peer 09a050703a1a4c0a8dfc04e4c2853797. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.4:46785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:44.555934 1629 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:44.685904 6438 raft_consensus.cc:482] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99e2f0f92de9442c9e73d00d03ca6b90)
I20240503 08:13:44.686120 6438 raft_consensus.cc:504] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:44.687072 6438 leader_election.cc:288] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
W20240503 08:13:44.688728 1922 leader_election.cc:334] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): Network error: Client connection negotiation failed: client connection to 127.24.123.2:39617: connect: Connection refused (error 111)
W20240503 08:13:44.688899 1921 leader_election.cc:334] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): Network error: Client connection negotiation failed: client connection to 127.24.123.1:41731: connect: Connection refused (error 111)
I20240503 08:13:44.689025 1921 leader_election.cc:302] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e615bd1588964e2fbe14f14f4aae0a98; no voters: 272c59640332489181cbf2cc52d266bb, 99e2f0f92de9442c9e73d00d03ca6b90
I20240503 08:13:44.689370 6438 raft_consensus.cc:2726] T 10646e4785b047d8a3ac6af5ddb06474 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:44.700829 6438 raft_consensus.cc:482] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99e2f0f92de9442c9e73d00d03ca6b90)
I20240503 08:13:44.700978 6438 raft_consensus.cc:504] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } }
I20240503 08:13:44.701669 6438 leader_election.cc:288] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035)
W20240503 08:13:44.702713 1921 leader_election.cc:334] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): Network error: Client connection negotiation failed: client connection to 127.24.123.1:41731: connect: Connection refused (error 111)
I20240503 08:13:44.705476 1589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ed99e3dd124de09353f6ed509f3eb1" candidate_uuid: "e615bd1588964e2fbe14f14f4aae0a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c6b280e8ecd54855b7b21b58a223149a" is_pre_election: true
I20240503 08:13:44.705709 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 1508
W20240503 08:13:44.726487 1921 connection.cc:535] server connection from 127.24.123.5:44015 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:44.727058 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 1640
W20240503 08:13:44.727875 1789 consensus_peers.cc:485] T 86b7e012f1544986a9460cd5dac5cece P f7f6592d13d245508261964ca6ffc5c4 -> Peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Couldn't send request to peer c6b280e8ecd54855b7b21b58a223149a. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.5:44035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:44.727931 1921 leader_election.cc:334] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Network error: Client connection negotiation failed: client connection to 127.24.123.5:44035: connect: Connection refused (error 111)
I20240503 08:13:44.728082 1921 leader_election.cc:302] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e615bd1588964e2fbe14f14f4aae0a98; no voters: 99e2f0f92de9442c9e73d00d03ca6b90, c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:44.728365 6438 raft_consensus.cc:2726] T b7ed99e3dd124de09353f6ed509f3eb1 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:44.732920 1763 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:44.802587 6438 raft_consensus.cc:482] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99e2f0f92de9442c9e73d00d03ca6b90)
I20240503 08:13:44.802825 6438 raft_consensus.cc:504] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09a050703a1a4c0a8dfc04e4c2853797" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 46785 } } peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:44.803714 6438 leader_election.cc:288] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785), 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731)
W20240503 08:13:44.805039 1922 leader_election.cc:334] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09a050703a1a4c0a8dfc04e4c2853797 (127.24.123.4:46785): Network error: Client connection negotiation failed: client connection to 127.24.123.4:46785: connect: Connection refused (error 111)
W20240503 08:13:44.805205 1921 leader_election.cc:334] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): Network error: Client connection negotiation failed: client connection to 127.24.123.1:41731: connect: Connection refused (error 111)
I20240503 08:13:44.805320 1921 leader_election.cc:302] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e615bd1588964e2fbe14f14f4aae0a98; no voters: 09a050703a1a4c0a8dfc04e4c2853797, 99e2f0f92de9442c9e73d00d03ca6b90
I20240503 08:13:44.805574 6438 raft_consensus.cc:2726] T b7996d398e3440ebbb8f942bda8f6833 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:44.838070 6131 raft_consensus.cc:482] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 272c59640332489181cbf2cc52d266bb)
I20240503 08:13:44.838315 6131 raft_consensus.cc:504] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } }
I20240503 08:13:44.839154 6131 leader_election.cc:288] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
W20240503 08:13:44.840653 1790 leader_election.cc:334] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): Network error: Client connection negotiation failed: client connection to 127.24.123.2:39617: connect: Connection refused (error 111)
W20240503 08:13:44.840832 1789 leader_election.cc:334] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): Network error: Client connection negotiation failed: client connection to 127.24.123.3:34237: connect: Connection refused (error 111)
I20240503 08:13:44.840975 1789 leader_election.cc:302] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7f6592d13d245508261964ca6ffc5c4; no voters: 272c59640332489181cbf2cc52d266bb, 71ad6bf07682497aaa1e9429320aef79
I20240503 08:13:44.841221 6131 raft_consensus.cc:2726] T 130b43c1e784400fb8851a71428014a1 P f7f6592d13d245508261964ca6ffc5c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:44.843254 6438 raft_consensus.cc:482] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 99e2f0f92de9442c9e73d00d03ca6b90)
I20240503 08:13:44.843402 6438 raft_consensus.cc:504] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:44.843983 6438 leader_election.cc:288] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237)
W20240503 08:13:44.845305 1921 leader_election.cc:334] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): Network error: Client connection negotiation failed: client connection to 127.24.123.1:41731: connect: Connection refused (error 111)
W20240503 08:13:44.845499 1921 leader_election.cc:334] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237): Network error: Client connection negotiation failed: client connection to 127.24.123.3:34237: connect: Connection refused (error 111)
I20240503 08:13:44.845583 1921 leader_election.cc:302] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e615bd1588964e2fbe14f14f4aae0a98; no voters: 71ad6bf07682497aaa1e9429320aef79, 99e2f0f92de9442c9e73d00d03ca6b90
I20240503 08:13:44.845808 6438 raft_consensus.cc:2726] T 27a469ffc17447ed86860e991ff3534a P e615bd1588964e2fbe14f14f4aae0a98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:13:44.866732 6438 raft_consensus.cc:482] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99e2f0f92de9442c9e73d00d03ca6b90)
I20240503 08:13:44.866923 6438 raft_consensus.cc:504] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99e2f0f92de9442c9e73d00d03ca6b90" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 41731 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } }
I20240503 08:13:44.867482 6438 leader_election.cc:288] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731), c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035)
W20240503 08:13:44.869065 1921 leader_election.cc:334] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99e2f0f92de9442c9e73d00d03ca6b90 (127.24.123.1:41731): Network error: Client connection negotiation failed: client connection to 127.24.123.1:41731: connect: Connection refused (error 111)
W20240503 08:13:44.869321 1921 leader_election.cc:334] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Network error: Client connection negotiation failed: client connection to 127.24.123.5:44035: connect: Connection refused (error 111)
I20240503 08:13:44.869486 1921 leader_election.cc:302] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e615bd1588964e2fbe14f14f4aae0a98; no voters: 99e2f0f92de9442c9e73d00d03ca6b90, c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:44.869733 6438 raft_consensus.cc:2726] T 2bd736780ea840ab9b6630a4acd80ae9 P e615bd1588964e2fbe14f14f4aae0a98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:44.888411 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 1640
I20240503 08:13:44.908773 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 1773
I20240503 08:13:44.914572 1896 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:45.057103 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 1773
I20240503 08:13:45.066462 6458 raft_consensus.cc:482] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 272c59640332489181cbf2cc52d266bb)
I20240503 08:13:45.066650 6458 raft_consensus.cc:504] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:45.068219 6458 leader_election.cc:288] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603)
W20240503 08:13:45.069409 2773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.2:39617: connect: Connection refused (error 111)
I20240503 08:13:45.070645 6458 raft_consensus.cc:482] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 272c59640332489181cbf2cc52d266bb)
I20240503 08:13:45.070830 6458 raft_consensus.cc:504] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "6c93e47973e047d0a162a50f72f9a757" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 36447 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:45.071532 6458 leader_election.cc:288] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447)
W20240503 08:13:45.072516 2773 leader_election.cc:334] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): Network error: Client connection negotiation failed: client connection to 127.24.123.2:39617: connect: Connection refused (error 111)
W20240503 08:13:45.073536 2773 leader_election.cc:334] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): Network error: Client connection negotiation failed: client connection to 127.24.123.2:39617: connect: Connection refused (error 111)
W20240503 08:13:45.073704 2773 leader_election.cc:334] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c93e47973e047d0a162a50f72f9a757 (127.24.123.6:36447): Network error: Client connection negotiation failed: client connection to 127.24.123.6:36447: connect: Connection refused (error 111)
I20240503 08:13:45.073909 2773 leader_election.cc:302] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 237c0b0fe44549a89d1b5bfc8c55da91; no voters: 272c59640332489181cbf2cc52d266bb, 6c93e47973e047d0a162a50f72f9a757
I20240503 08:13:45.074231 6458 raft_consensus.cc:2726] T f9541e40695844a1b05ca1bc0dd8a10d P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:45.079275 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 1905
W20240503 08:13:45.079854 6459 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:45.068513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.123.7:33603 (local address 127.24.123.9:44137)
0503 08:13:45.069329 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:45.069346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:45.069491 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:45.079071 (+ 9580us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.123.7:33603: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":248,"threads_started":1}
W20240503 08:13:45.080763 2774 leader_election.cc:334] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603): Network error: Client connection negotiation failed: client connection to 127.24.123.7:33603: connect: Connection refused (error 111)
I20240503 08:13:45.080884 2774 leader_election.cc:302] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 237c0b0fe44549a89d1b5bfc8c55da91; no voters: 272c59640332489181cbf2cc52d266bb, f7f6592d13d245508261964ca6ffc5c4
I20240503 08:13:45.081136 6458 raft_consensus.cc:2726] T 400a0317f9184c99a449f7f160fa998f P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:45.085170 2027 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:45.198953 6458 raft_consensus.cc:482] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 272c59640332489181cbf2cc52d266bb)
I20240503 08:13:45.199165 6458 raft_consensus.cc:504] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "e615bd1588964e2fbe14f14f4aae0a98" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43749 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:45.199992 6458 leader_election.cc:288] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749)
W20240503 08:13:45.201274 2773 leader_election.cc:334] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): Network error: Client connection negotiation failed: client connection to 127.24.123.2:39617: connect: Connection refused (error 111)
I20240503 08:13:45.231016 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 1905
I20240503 08:13:45.239904 6458 raft_consensus.cc:482] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 272c59640332489181cbf2cc52d266bb)
I20240503 08:13:45.240114 6458 raft_consensus.cc:504] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "272c59640332489181cbf2cc52d266bb" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 39617 } } peers { permanent_uuid: "c6b280e8ecd54855b7b21b58a223149a" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 44035 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:45.240877 6458 leader_election.cc:288] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617), c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035)
W20240503 08:13:45.242026 2773 leader_election.cc:334] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 272c59640332489181cbf2cc52d266bb (127.24.123.2:39617): Network error: Client connection negotiation failed: client connection to 127.24.123.2:39617: connect: Connection refused (error 111)
W20240503 08:13:45.242189 2772 leader_election.cc:334] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c6b280e8ecd54855b7b21b58a223149a (127.24.123.5:44035): Network error: Client connection negotiation failed: client connection to 127.24.123.5:44035: connect: Connection refused (error 111)
I20240503 08:13:45.242336 2772 leader_election.cc:302] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 237c0b0fe44549a89d1b5bfc8c55da91; no voters: 272c59640332489181cbf2cc52d266bb, c6b280e8ecd54855b7b21b58a223149a
I20240503 08:13:45.242586 6458 raft_consensus.cc:2726] T d9151dcb76174b27991fdcf3839db0a5 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:45.252053 2774 connection.cc:535] client connection to 127.24.123.8:43749 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:45.252406 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 2662
W20240503 08:13:45.253002 2774 leader_election.cc:334] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e615bd1588964e2fbe14f14f4aae0a98 (127.24.123.8:43749): Network error: Client connection negotiation failed: client connection to 127.24.123.8:43749: connect: Connection refused (error 111)
I20240503 08:13:45.253159 2774 leader_election.cc:302] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 237c0b0fe44549a89d1b5bfc8c55da91; no voters: 272c59640332489181cbf2cc52d266bb, e615bd1588964e2fbe14f14f4aae0a98
I20240503 08:13:45.253573 6458 raft_consensus.cc:2726] T 7cc8a3241af248578b6ce87b1e33cea9 P 237c0b0fe44549a89d1b5bfc8c55da91 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 08:13:45.258394 2884 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:45.383265 6458 raft_consensus.cc:482] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71ad6bf07682497aaa1e9429320aef79)
I20240503 08:13:45.383483 6458 raft_consensus.cc:504] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71ad6bf07682497aaa1e9429320aef79" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 34237 } } peers { permanent_uuid: "f7f6592d13d245508261964ca6ffc5c4" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 33603 } attrs { promote: false } } peers { permanent_uuid: "237c0b0fe44549a89d1b5bfc8c55da91" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 41169 } attrs { promote: false } }
I20240503 08:13:45.384172 6458 leader_election.cc:288] T af5c3865ca694bf090dce19f1100e5eb P 237c0b0fe44549a89d1b5bfc8c55da91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71ad6bf07682497aaa1e9429320aef79 (127.24.123.3:34237), f7f6592d13d245508261964ca6ffc5c4 (127.24.123.7:33603)
I20240503 08:13:45.384373 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 2662
I20240503 08:13:45.402678 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 31969
I20240503 08:13:45.403683 32029 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:45.594066 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 31969
2024-05-03T08:13:45Z chronyd exiting
[ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (55076 ms)
[----------] 1 test from TserverAddedDuringRebalancingTest (55077 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5
I20240503 08:13:45.734066 25068 test_util.cc:268] Using random seed: 1282478991
I20240503 08:13:45.735821 25068 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:13:45.735891 25068 ts_itest-base.cc:113] --------------
I20240503 08:13:45.735922 25068 ts_itest-base.cc:114] 4 tablet servers
I20240503 08:13:45.735982 25068 ts_itest-base.cc:115] 1 replicas per TS
I20240503 08:13:45.736027 25068 ts_itest-base.cc:116] --------------
2024-05-03T08:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:45Z Disabled control of system clock
I20240503 08:13:45.777539 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:40289
--webserver_interface=127.24.123.62
--webserver_port=0
--builtin_ntp_servers=127.24.123.20:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.123.62:40289
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 08:13:46.091353 6474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:46.091626 6474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:46.091713 6474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:46.098676 6474 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:46.098779 6474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:46.098832 6474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:46.098908 6474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:46.098963 6474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:46.107424 6474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.123.62:40289
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:40289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.123.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:46.108505 6474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:46.110054 6474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:46.119868 6479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:46.120280 6480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:46.120682 6482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:46.121613 6474 server_base.cc:1034] running on GCE node
I20240503 08:13:46.122232 6474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:46.123502 6474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:46.124722 6474 hybrid_clock.cc:648] HybridClock initialized: now 1714724026124670 us; error 39 us; skew 500 ppm
I20240503 08:13:46.125038 6474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:46.127061 6474 webserver.cc:460] Webserver started at http://127.24.123.62:32775/ using document root <none> and password file <none>
I20240503 08:13:46.127718 6474 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:46.127820 6474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:46.128090 6474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:46.130470 6474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "df6fa78cc214403eac8aa0c3c4706f68"
format_stamp: "Formatted at 2024-05-03 08:13:46 on dist-test-slave-gkrw"
I20240503 08:13:46.131201 6474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "df6fa78cc214403eac8aa0c3c4706f68"
format_stamp: "Formatted at 2024-05-03 08:13:46 on dist-test-slave-gkrw"
I20240503 08:13:46.136080 6474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20240503 08:13:46.139107 6488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:46.140280 6474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:46.140426 6474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
uuid: "df6fa78cc214403eac8aa0c3c4706f68"
format_stamp: "Formatted at 2024-05-03 08:13:46 on dist-test-slave-gkrw"
I20240503 08:13:46.140616 6474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:46.165244 6474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:46.166019 6474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:46.166266 6474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:46.182833 6474 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.62:40289
I20240503 08:13:46.182852 6539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.62:40289 every 8 connection(s)
I20240503 08:13:46.184119 6474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:13:46.187645 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:46.191315 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 6474
I20240503 08:13:46.191552 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:13:46.197808 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68: Bootstrap starting.
I20240503 08:13:46.201133 6540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:46.202301 6540 log.cc:824] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:46.205011 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68: No bootstrap required, opened a new log
I20240503 08:13:46.210767 6540 raft_consensus.cc:348] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6fa78cc214403eac8aa0c3c4706f68" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40289 } }
I20240503 08:13:46.211109 6540 raft_consensus.cc:374] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:46.211201 6540 raft_consensus.cc:729] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6fa78cc214403eac8aa0c3c4706f68, State: Initialized, Role: FOLLOWER
I20240503 08:13:46.211817 6540 consensus_queue.cc:260] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6fa78cc214403eac8aa0c3c4706f68" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40289 } }
I20240503 08:13:46.211990 6540 raft_consensus.cc:388] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:46.212080 6540 raft_consensus.cc:482] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:46.212204 6540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:46.214073 6540 raft_consensus.cc:504] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6fa78cc214403eac8aa0c3c4706f68" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40289 } }
I20240503 08:13:46.214561 6540 leader_election.cc:302] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df6fa78cc214403eac8aa0c3c4706f68; no voters:
I20240503 08:13:46.215106 6540 leader_election.cc:288] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:46.215219 6545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:46.215821 6545 raft_consensus.cc:686] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [term 1 LEADER]: Becoming Leader. State: Replica: df6fa78cc214403eac8aa0c3c4706f68, State: Running, Role: LEADER
I20240503 08:13:46.216270 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6fa78cc214403eac8aa0c3c4706f68" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40289 } }
I20240503 08:13:46.216818 6540 sys_catalog.cc:564] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:46.218891 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df6fa78cc214403eac8aa0c3c4706f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6fa78cc214403eac8aa0c3c4706f68" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40289 } } }
I20240503 08:13:46.218930 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6fa78cc214403eac8aa0c3c4706f68. Latest consensus state: current_term: 1 leader_uuid: "df6fa78cc214403eac8aa0c3c4706f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6fa78cc214403eac8aa0c3c4706f68" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 40289 } } }
I20240503 08:13:46.219214 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:46.219219 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:46.220808 6551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:46.225047 6551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:46.232076 6551 catalog_manager.cc:1347] Generated new cluster ID: f87f2ae5eb394620b4a16b39d3f01767
I20240503 08:13:46.232182 6551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:13:46.246639 6551 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:46.247972 6551 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:46.254582 6551 catalog_manager.cc:5857] T 00000000000000000000000000000000 P df6fa78cc214403eac8aa0c3c4706f68: Generated new TSK 0
I20240503 08:13:46.255342 6551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:13:46.262310 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.1:0
--local_ip_for_outbound_sockets=127.24.123.1
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40289
--builtin_ntp_servers=127.24.123.20:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:13:46.575821 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:46.576090 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:46.576201 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:46.583365 6564 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:46.583492 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:46.583639 6564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.1
I20240503 08:13:46.592098 6564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.1
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:46.593307 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:46.594959 6564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:46.605926 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:46.606388 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:46.606896 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:46.607509 6564 server_base.cc:1034] running on GCE node
I20240503 08:13:46.608008 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:46.608794 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:46.610009 6564 hybrid_clock.cc:648] HybridClock initialized: now 1714724026609967 us; error 29 us; skew 500 ppm
I20240503 08:13:46.610298 6564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:46.612610 6564 webserver.cc:460] Webserver started at http://127.24.123.1:38689/ using document root <none> and password file <none>
I20240503 08:13:46.613320 6564 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:46.613451 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:46.613762 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:46.616349 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "547814138c6346d180edab80d3927f83"
format_stamp: "Formatted at 2024-05-03 08:13:46 on dist-test-slave-gkrw"
I20240503 08:13:46.617164 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "547814138c6346d180edab80d3927f83"
format_stamp: "Formatted at 2024-05-03 08:13:46 on dist-test-slave-gkrw"
I20240503 08:13:46.622285 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 08:13:46.625530 6578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:46.626739 6564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:46.626916 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "547814138c6346d180edab80d3927f83"
format_stamp: "Formatted at 2024-05-03 08:13:46 on dist-test-slave-gkrw"
I20240503 08:13:46.627101 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:46.657511 6564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:46.658336 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:46.658603 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:46.659547 6564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:46.661001 6564 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:46.661092 6564 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:46.661202 6564 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:46.661273 6564 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:46.689018 6564 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.1:40801
I20240503 08:13:46.689077 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.1:40801 every 8 connection(s)
I20240503 08:13:46.690217 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:13:46.690754 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 6564
I20240503 08:13:46.690939 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:13:46.694283 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.2:0
--local_ip_for_outbound_sockets=127.24.123.2
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40289
--builtin_ntp_servers=127.24.123.20:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:46.702033 6691 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40289
I20240503 08:13:46.702280 6691 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:46.703030 6691 heartbeater.cc:511] Master 127.24.123.62:40289 requested a full tablet report, sending...
I20240503 08:13:46.704994 6505 ts_manager.cc:194] Registered new tserver with Master: 547814138c6346d180edab80d3927f83 (127.24.123.1:40801)
I20240503 08:13:46.706387 6505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.1:38265
W20240503 08:13:47.008605 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:47.008878 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:47.009003 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:47.016232 6694 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:47.016346 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:47.016505 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.2
I20240503 08:13:47.025254 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.2
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:47.026415 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:47.028012 6694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:47.039095 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.039541 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.039907 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:47.040567 6694 server_base.cc:1034] running on GCE node
I20240503 08:13:47.041054 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:47.041843 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:47.043025 6694 hybrid_clock.cc:648] HybridClock initialized: now 1714724027042988 us; error 31 us; skew 500 ppm
I20240503 08:13:47.043283 6694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:47.049016 6694 webserver.cc:460] Webserver started at http://127.24.123.2:34903/ using document root <none> and password file <none>
I20240503 08:13:47.049679 6694 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:47.049785 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:47.050037 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:47.052453 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d2c9b8623c324170b7161534f65be955"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:47.053126 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d2c9b8623c324170b7161534f65be955"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:47.057790 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240503 08:13:47.060760 6708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.061928 6694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:47.062062 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d2c9b8623c324170b7161534f65be955"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:47.062228 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:47.089578 6694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:47.090332 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:47.090574 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:47.091511 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:47.092914 6694 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:47.093003 6694 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.093101 6694 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:47.093166 6694 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.120234 6694 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.2:33949
I20240503 08:13:47.120301 6820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.2:33949 every 8 connection(s)
I20240503 08:13:47.121381 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:13:47.130007 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 6694
I20240503 08:13:47.130192 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:13:47.132390 6821 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40289
I20240503 08:13:47.132668 6821 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:47.133487 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.3:0
--local_ip_for_outbound_sockets=127.24.123.3
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40289
--builtin_ntp_servers=127.24.123.20:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:47.133570 6821 heartbeater.cc:511] Master 127.24.123.62:40289 requested a full tablet report, sending...
I20240503 08:13:47.134857 6505 ts_manager.cc:194] Registered new tserver with Master: d2c9b8623c324170b7161534f65be955 (127.24.123.2:33949)
I20240503 08:13:47.135792 6505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.2:54021
W20240503 08:13:47.441207 6824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:47.441457 6824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:47.441577 6824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:47.448685 6824 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:47.448782 6824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:47.448928 6824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.3
I20240503 08:13:47.457401 6824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.3
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:47.458532 6824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:47.460144 6824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:47.471521 6829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.472000 6830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.472208 6832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:47.473086 6824 server_base.cc:1034] running on GCE node
I20240503 08:13:47.473587 6824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:47.474403 6824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:47.475683 6824 hybrid_clock.cc:648] HybridClock initialized: now 1714724027475627 us; error 41 us; skew 500 ppm
I20240503 08:13:47.475983 6824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:47.478211 6824 webserver.cc:460] Webserver started at http://127.24.123.3:37871/ using document root <none> and password file <none>
I20240503 08:13:47.478973 6824 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:47.479100 6824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:47.479401 6824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:47.481895 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2433183a6f5046e8ba9b00190b1440a8"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:47.482648 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2433183a6f5046e8ba9b00190b1440a8"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:47.487460 6824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:13:47.490391 6838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.491560 6824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:47.491693 6824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2433183a6f5046e8ba9b00190b1440a8"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:47.491865 6824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:47.518082 6824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:47.518815 6824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:47.519078 6824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:47.519897 6824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:47.521220 6824 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:47.521313 6824 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.521412 6824 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:47.521477 6824 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.548957 6824 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.3:33081
I20240503 08:13:47.549018 6950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.3:33081 every 8 connection(s)
I20240503 08:13:47.550176 6824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:13:47.558375 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 6824
I20240503 08:13:47.558552 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:13:47.561981 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.4:0
--local_ip_for_outbound_sockets=127.24.123.4
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40289
--builtin_ntp_servers=127.24.123.20:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:47.562047 6951 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40289
I20240503 08:13:47.562314 6951 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:47.563144 6951 heartbeater.cc:511] Master 127.24.123.62:40289 requested a full tablet report, sending...
I20240503 08:13:47.564424 6505 ts_manager.cc:194] Registered new tserver with Master: 2433183a6f5046e8ba9b00190b1440a8 (127.24.123.3:33081)
I20240503 08:13:47.565182 6505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.3:33959
I20240503 08:13:47.708890 6691 heartbeater.cc:503] Master 127.24.123.62:40289 was elected leader, sending a full tablet report...
W20240503 08:13:47.871423 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:47.871677 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:47.871802 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:47.879029 6954 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:47.879150 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:47.879300 6954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.4
I20240503 08:13:47.888440 6954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.4
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:47.889652 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:47.891278 6954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:47.902343 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.902905 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.903246 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:47.904076 6954 server_base.cc:1034] running on GCE node
I20240503 08:13:47.904599 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:47.905429 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:47.906633 6954 hybrid_clock.cc:648] HybridClock initialized: now 1714724027906578 us; error 44 us; skew 500 ppm
I20240503 08:13:47.906956 6954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:47.908962 6954 webserver.cc:460] Webserver started at http://127.24.123.4:34621/ using document root <none> and password file <none>
I20240503 08:13:47.909626 6954 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:47.909732 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:47.909992 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:47.912463 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2e330bc06a8444e983e94680338fff0d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:47.913157 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2e330bc06a8444e983e94680338fff0d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:47.918061 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 08:13:47.921350 6968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.922639 6954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:47.922784 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2e330bc06a8444e983e94680338fff0d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:47.922994 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:47.938915 6954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:47.939705 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:47.939957 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:47.940829 6954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:47.942193 6954 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:47.942276 6954 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.942373 6954 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:47.942440 6954 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.970330 6954 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.4:33371
I20240503 08:13:47.970378 7080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.4:33371 every 8 connection(s)
I20240503 08:13:47.971539 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:13:47.976897 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 6954
I20240503 08:13:47.977100 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:13:47.983134 7081 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40289
I20240503 08:13:47.983387 7081 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:47.984210 7081 heartbeater.cc:511] Master 127.24.123.62:40289 requested a full tablet report, sending...
I20240503 08:13:47.985612 6505 ts_manager.cc:194] Registered new tserver with Master: 2e330bc06a8444e983e94680338fff0d (127.24.123.4:33371)
I20240503 08:13:47.986609 6505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.4:35781
I20240503 08:13:47.992237 25068 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:13:48.006031 6505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50100:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:13:48.031004 7016 tablet_service.cc:1467] Processing CreateTablet for tablet 824c51fd02104b6589716d191b26f31a (DEFAULT_TABLE table=TestTable [id=b0bb448773544c78a4cca2592ead12e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:48.032202 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824c51fd02104b6589716d191b26f31a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:48.041376 7095 tablet_bootstrap.cc:492] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Bootstrap starting.
I20240503 08:13:48.044106 7095 tablet_bootstrap.cc:654] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:48.045045 7095 log.cc:824] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:48.047267 7095 tablet_bootstrap.cc:492] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: No bootstrap required, opened a new log
I20240503 08:13:48.047502 7095 ts_tablet_manager.cc:1367] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 08:13:48.053464 7095 raft_consensus.cc:348] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } }
I20240503 08:13:48.053794 7095 raft_consensus.cc:374] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:48.053884 7095 raft_consensus.cc:729] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e330bc06a8444e983e94680338fff0d, State: Initialized, Role: FOLLOWER
I20240503 08:13:48.054435 7095 consensus_queue.cc:260] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } }
I20240503 08:13:48.054630 7095 raft_consensus.cc:388] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:48.054742 7095 raft_consensus.cc:482] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:48.054891 7095 raft_consensus.cc:3037] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:48.056779 7095 raft_consensus.cc:504] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } }
I20240503 08:13:48.057207 7095 leader_election.cc:302] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e330bc06a8444e983e94680338fff0d; no voters:
I20240503 08:13:48.057719 7095 leader_election.cc:288] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:48.057842 7097 raft_consensus.cc:2781] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:48.058676 7081 heartbeater.cc:503] Master 127.24.123.62:40289 was elected leader, sending a full tablet report...
I20240503 08:13:48.058769 7097 raft_consensus.cc:686] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 1 LEADER]: Becoming Leader. State: Replica: 2e330bc06a8444e983e94680338fff0d, State: Running, Role: LEADER
I20240503 08:13:48.059264 7097 consensus_queue.cc:237] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } }
I20240503 08:13:48.059880 7095 ts_tablet_manager.cc:1398] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20240503 08:13:48.063987 6505 catalog_manager.cc:5484] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e330bc06a8444e983e94680338fff0d (127.24.123.4). New cstate: current_term: 1 leader_uuid: "2e330bc06a8444e983e94680338fff0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:48.073640 25068 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:13:48.075299 25068 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 547814138c6346d180edab80d3927f83 to finish bootstrapping
I20240503 08:13:48.083741 25068 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d2c9b8623c324170b7161534f65be955 to finish bootstrapping
I20240503 08:13:48.091130 25068 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2433183a6f5046e8ba9b00190b1440a8 to finish bootstrapping
I20240503 08:13:48.098526 25068 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2e330bc06a8444e983e94680338fff0d to finish bootstrapping
I20240503 08:13:48.105137 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 6824
I20240503 08:13:48.105913 6946 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:48.138218 6821 heartbeater.cc:503] Master 127.24.123.62:40289 was elected leader, sending a full tablet report...
I20240503 08:13:48.191692 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 6824
I20240503 08:13:48.208362 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 6954
I20240503 08:13:48.209357 7076 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:48.292286 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 6954
I20240503 08:13:52.056881 6505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50100:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:52.074612 6626 tablet_service.cc:1467] Processing CreateTablet for tablet 48656f3b08464e33bdb78b47bc5a796f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6051da82681547ebaa3b849bfd5625b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:52.074956 6756 tablet_service.cc:1467] Processing CreateTablet for tablet 9cda1f07ae62489b82b2e8ea4e7c26ee (DEFAULT_TABLE table=rebalance_test_table_0 [id=6051da82681547ebaa3b849bfd5625b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:52.074653 6625 tablet_service.cc:1467] Processing CreateTablet for tablet aef386c1619146d7a9da637f112899c6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6051da82681547ebaa3b849bfd5625b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:52.076004 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48656f3b08464e33bdb78b47bc5a796f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:52.076191 6756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cda1f07ae62489b82b2e8ea4e7c26ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:52.076457 6625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef386c1619146d7a9da637f112899c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:52.086087 7116 tablet_bootstrap.cc:492] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955: Bootstrap starting.
I20240503 08:13:52.087872 7117 tablet_bootstrap.cc:492] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83: Bootstrap starting.
I20240503 08:13:52.089092 7116 tablet_bootstrap.cc:654] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:52.090394 7117 tablet_bootstrap.cc:654] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:52.090502 7116 log.cc:824] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:52.091342 7117 log.cc:824] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:52.092936 7116 tablet_bootstrap.cc:492] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955: No bootstrap required, opened a new log
I20240503 08:13:52.093156 7116 ts_tablet_manager.cc:1367] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240503 08:13:52.093488 7117 tablet_bootstrap.cc:492] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83: No bootstrap required, opened a new log
I20240503 08:13:52.093725 7117 ts_tablet_manager.cc:1367] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:13:52.099589 7116 raft_consensus.cc:348] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.099694 7117 raft_consensus.cc:348] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.099898 7116 raft_consensus.cc:374] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:52.099984 7117 raft_consensus.cc:374] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:52.099992 7116 raft_consensus.cc:729] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c9b8623c324170b7161534f65be955, State: Initialized, Role: FOLLOWER
I20240503 08:13:52.100068 7117 raft_consensus.cc:729] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547814138c6346d180edab80d3927f83, State: Initialized, Role: FOLLOWER
I20240503 08:13:52.100559 7116 consensus_queue.cc:260] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.100610 7117 consensus_queue.cc:260] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.100736 7116 raft_consensus.cc:388] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:52.100811 7117 raft_consensus.cc:388] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:52.100863 7116 raft_consensus.cc:482] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:52.100931 7117 raft_consensus.cc:482] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:52.101015 7116 raft_consensus.cc:3037] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:52.101078 7117 raft_consensus.cc:3037] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:52.102907 7116 raft_consensus.cc:504] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.102946 7117 raft_consensus.cc:504] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.103327 7116 leader_election.cc:302] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c9b8623c324170b7161534f65be955; no voters:
I20240503 08:13:52.103328 7117 leader_election.cc:302] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547814138c6346d180edab80d3927f83; no voters:
I20240503 08:13:52.103910 7116 leader_election.cc:288] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:52.103938 7117 leader_election.cc:288] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:52.104032 7120 raft_consensus.cc:2781] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:52.104069 7121 raft_consensus.cc:2781] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:52.109642 7120 raft_consensus.cc:686] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [term 1 LEADER]: Becoming Leader. State: Replica: d2c9b8623c324170b7161534f65be955, State: Running, Role: LEADER
I20240503 08:13:52.110100 7116 ts_tablet_manager.cc:1398] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955: Time spent starting tablet: real 0.017s user 0.007s sys 0.011s
I20240503 08:13:52.110249 7120 consensus_queue.cc:237] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.111650 7121 raft_consensus.cc:686] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [term 1 LEADER]: Becoming Leader. State: Replica: 547814138c6346d180edab80d3927f83, State: Running, Role: LEADER
I20240503 08:13:52.111891 7117 ts_tablet_manager.cc:1398] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20240503 08:13:52.112181 7121 consensus_queue.cc:237] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.113579 7117 tablet_bootstrap.cc:492] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83: Bootstrap starting.
I20240503 08:13:52.115581 7117 tablet_bootstrap.cc:654] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:52.115443 6505 catalog_manager.cc:5484] T 9cda1f07ae62489b82b2e8ea4e7c26ee P d2c9b8623c324170b7161534f65be955 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2c9b8623c324170b7161534f65be955 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "d2c9b8623c324170b7161534f65be955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:52.119560 7117 tablet_bootstrap.cc:492] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83: No bootstrap required, opened a new log
I20240503 08:13:52.119743 7117 ts_tablet_manager.cc:1367] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240503 08:13:52.120750 7117 raft_consensus.cc:348] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.120950 7117 raft_consensus.cc:374] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:52.121062 7117 raft_consensus.cc:729] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547814138c6346d180edab80d3927f83, State: Initialized, Role: FOLLOWER
I20240503 08:13:52.121290 7117 consensus_queue.cc:260] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.121491 7117 raft_consensus.cc:388] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:52.121577 7117 raft_consensus.cc:482] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:52.121704 7117 raft_consensus.cc:3037] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:52.121528 6504 catalog_manager.cc:5484] T 48656f3b08464e33bdb78b47bc5a796f P 547814138c6346d180edab80d3927f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 547814138c6346d180edab80d3927f83 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "547814138c6346d180edab80d3927f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:52.124501 7117 raft_consensus.cc:504] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.124729 7117 leader_election.cc:302] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547814138c6346d180edab80d3927f83; no voters:
I20240503 08:13:52.124907 7117 leader_election.cc:288] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:52.124969 7121 raft_consensus.cc:2781] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:52.125134 7121 raft_consensus.cc:686] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [term 1 LEADER]: Becoming Leader. State: Replica: 547814138c6346d180edab80d3927f83, State: Running, Role: LEADER
I20240503 08:13:52.125337 7121 consensus_queue.cc:237] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.126080 7117 ts_tablet_manager.cc:1398] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240503 08:13:52.127918 6504 catalog_manager.cc:5484] T aef386c1619146d7a9da637f112899c6 P 547814138c6346d180edab80d3927f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 547814138c6346d180edab80d3927f83 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "547814138c6346d180edab80d3927f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:52.612674 6504 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50100:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:52.619730 6755 tablet_service.cc:1467] Processing CreateTablet for tablet 0b33dc0158fe4cc5b9966a633c2d3170 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a174bf50c674ebe8a1a03327b68ccf6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:52.619686 6756 tablet_service.cc:1467] Processing CreateTablet for tablet deb7075982b544f39fafddb2cfb41c82 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a174bf50c674ebe8a1a03327b68ccf6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:52.620168 6756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb7075982b544f39fafddb2cfb41c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:52.620383 6755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b33dc0158fe4cc5b9966a633c2d3170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:52.620546 6626 tablet_service.cc:1467] Processing CreateTablet for tablet 1bca5ec65f5f4901947eb527a90b2244 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3a174bf50c674ebe8a1a03327b68ccf6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:52.620937 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bca5ec65f5f4901947eb527a90b2244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:52.625164 7117 tablet_bootstrap.cc:492] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83: Bootstrap starting.
I20240503 08:13:52.626472 7152 tablet_bootstrap.cc:492] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955: Bootstrap starting.
I20240503 08:13:52.626745 7117 tablet_bootstrap.cc:654] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:52.628175 7152 tablet_bootstrap.cc:654] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:52.628883 7117 tablet_bootstrap.cc:492] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83: No bootstrap required, opened a new log
I20240503 08:13:52.629050 7117 ts_tablet_manager.cc:1367] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:13:52.629730 7117 raft_consensus.cc:348] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.629930 7117 raft_consensus.cc:374] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:52.630023 7117 raft_consensus.cc:729] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547814138c6346d180edab80d3927f83, State: Initialized, Role: FOLLOWER
I20240503 08:13:52.630048 7152 tablet_bootstrap.cc:492] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955: No bootstrap required, opened a new log
I20240503 08:13:52.630174 7152 ts_tablet_manager.cc:1367] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:52.630275 7117 consensus_queue.cc:260] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.630453 7117 raft_consensus.cc:388] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:52.630561 7117 raft_consensus.cc:482] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:52.630661 7117 raft_consensus.cc:3037] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:52.631187 7152 raft_consensus.cc:348] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.631448 7152 raft_consensus.cc:374] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:52.631548 7152 raft_consensus.cc:729] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c9b8623c324170b7161534f65be955, State: Initialized, Role: FOLLOWER
I20240503 08:13:52.631793 7152 consensus_queue.cc:260] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.631987 7152 raft_consensus.cc:388] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:52.632069 7152 raft_consensus.cc:482] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:52.632179 7152 raft_consensus.cc:3037] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:52.633270 7117 raft_consensus.cc:504] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.633523 7117 leader_election.cc:302] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547814138c6346d180edab80d3927f83; no voters:
I20240503 08:13:52.633751 7117 leader_election.cc:288] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:52.633833 7121 raft_consensus.cc:2781] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:52.634706 7152 raft_consensus.cc:504] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.635042 7121 raft_consensus.cc:686] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [term 1 LEADER]: Becoming Leader. State: Replica: 547814138c6346d180edab80d3927f83, State: Running, Role: LEADER
I20240503 08:13:52.635025 7152 leader_election.cc:302] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c9b8623c324170b7161534f65be955; no voters:
I20240503 08:13:52.635196 7117 ts_tablet_manager.cc:1398] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 08:13:52.635278 7121 consensus_queue.cc:237] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:52.635562 7152 leader_election.cc:288] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:52.635711 7157 raft_consensus.cc:2781] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:52.636336 7152 ts_tablet_manager.cc:1398] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 08:13:52.636643 7152 tablet_bootstrap.cc:492] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955: Bootstrap starting.
I20240503 08:13:52.638399 7152 tablet_bootstrap.cc:654] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:52.639771 7157 raft_consensus.cc:686] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [term 1 LEADER]: Becoming Leader. State: Replica: d2c9b8623c324170b7161534f65be955, State: Running, Role: LEADER
I20240503 08:13:52.640161 7157 consensus_queue.cc:237] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.640544 7152 tablet_bootstrap.cc:492] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955: No bootstrap required, opened a new log
I20240503 08:13:52.640800 7152 ts_tablet_manager.cc:1367] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:52.641570 7152 raft_consensus.cc:348] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.641716 7152 raft_consensus.cc:374] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:52.641800 7152 raft_consensus.cc:729] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c9b8623c324170b7161534f65be955, State: Initialized, Role: FOLLOWER
I20240503 08:13:52.642037 7152 consensus_queue.cc:260] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.642204 7152 raft_consensus.cc:388] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:52.642345 7152 raft_consensus.cc:482] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:52.642630 7152 raft_consensus.cc:3037] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:52.644263 6505 catalog_manager.cc:5484] T deb7075982b544f39fafddb2cfb41c82 P d2c9b8623c324170b7161534f65be955 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2c9b8623c324170b7161534f65be955 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "d2c9b8623c324170b7161534f65be955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:52.644212 6503 catalog_manager.cc:5484] T 1bca5ec65f5f4901947eb527a90b2244 P 547814138c6346d180edab80d3927f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 547814138c6346d180edab80d3927f83 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "547814138c6346d180edab80d3927f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:52.645330 7152 raft_consensus.cc:504] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.645560 7152 leader_election.cc:302] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c9b8623c324170b7161534f65be955; no voters:
I20240503 08:13:52.645787 7152 leader_election.cc:288] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:52.645861 7160 raft_consensus.cc:2781] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:52.646097 7160 raft_consensus.cc:686] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [term 1 LEADER]: Becoming Leader. State: Replica: d2c9b8623c324170b7161534f65be955, State: Running, Role: LEADER
I20240503 08:13:52.646417 7160 consensus_queue.cc:237] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:52.647837 7152 ts_tablet_manager.cc:1398] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240503 08:13:52.650092 6503 catalog_manager.cc:5484] T 0b33dc0158fe4cc5b9966a633c2d3170 P d2c9b8623c324170b7161534f65be955 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2c9b8623c324170b7161534f65be955 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "d2c9b8623c324170b7161534f65be955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:53.116247 6503 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50100:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:53.123663 6626 tablet_service.cc:1467] Processing CreateTablet for tablet 692e3e6b1f5d4e27b1bc80526338ae68 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8edc231e2bf94c10b6b5fe54222a1807]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:53.123833 6755 tablet_service.cc:1467] Processing CreateTablet for tablet 63a7ddcb883940ea934f235e1b3d8186 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8edc231e2bf94c10b6b5fe54222a1807]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:53.124060 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 692e3e6b1f5d4e27b1bc80526338ae68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:53.124236 6755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a7ddcb883940ea934f235e1b3d8186. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:53.127776 6625 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd0bda8e4ae4d9fb2ee058acf5479a8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8edc231e2bf94c10b6b5fe54222a1807]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:53.128122 6625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd0bda8e4ae4d9fb2ee058acf5479a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:53.128437 7117 tablet_bootstrap.cc:492] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83: Bootstrap starting.
I20240503 08:13:53.129985 7117 tablet_bootstrap.cc:654] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:53.131325 7152 tablet_bootstrap.cc:492] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955: Bootstrap starting.
I20240503 08:13:53.132905 7152 tablet_bootstrap.cc:654] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:53.134742 7117 tablet_bootstrap.cc:492] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83: No bootstrap required, opened a new log
I20240503 08:13:53.134908 7117 ts_tablet_manager.cc:1367] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:13:53.135540 7117 raft_consensus.cc:348] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.135666 7117 raft_consensus.cc:374] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:53.135746 7117 raft_consensus.cc:729] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547814138c6346d180edab80d3927f83, State: Initialized, Role: FOLLOWER
I20240503 08:13:53.135941 7117 consensus_queue.cc:260] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.136085 7117 raft_consensus.cc:388] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:53.136157 7117 raft_consensus.cc:482] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:53.136258 7117 raft_consensus.cc:3037] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:53.137959 7117 raft_consensus.cc:504] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.138151 7117 leader_election.cc:302] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547814138c6346d180edab80d3927f83; no voters:
I20240503 08:13:53.138309 7117 leader_election.cc:288] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:53.138388 7121 raft_consensus.cc:2781] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:53.139219 7152 tablet_bootstrap.cc:492] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955: No bootstrap required, opened a new log
I20240503 08:13:53.139349 7152 ts_tablet_manager.cc:1367] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 08:13:53.139914 7152 raft_consensus.cc:348] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.140120 7152 raft_consensus.cc:374] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:53.140229 7152 raft_consensus.cc:729] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c9b8623c324170b7161534f65be955, State: Initialized, Role: FOLLOWER
I20240503 08:13:53.140462 7152 consensus_queue.cc:260] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.140614 7152 raft_consensus.cc:388] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:53.140692 7152 raft_consensus.cc:482] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:53.140797 7152 raft_consensus.cc:3037] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:53.142500 7121 raft_consensus.cc:686] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [term 1 LEADER]: Becoming Leader. State: Replica: 547814138c6346d180edab80d3927f83, State: Running, Role: LEADER
I20240503 08:13:53.142486 7152 raft_consensus.cc:504] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.142695 7152 leader_election.cc:302] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c9b8623c324170b7161534f65be955; no voters:
I20240503 08:13:53.142740 7121 consensus_queue.cc:237] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.142859 7152 leader_election.cc:288] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:53.142856 7117 ts_tablet_manager.cc:1398] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:13:53.142942 7160 raft_consensus.cc:2781] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:53.143095 7160 raft_consensus.cc:686] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [term 1 LEADER]: Becoming Leader. State: Replica: d2c9b8623c324170b7161534f65be955, State: Running, Role: LEADER
I20240503 08:13:53.143173 7117 tablet_bootstrap.cc:492] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83: Bootstrap starting.
I20240503 08:13:53.143292 7160 consensus_queue.cc:237] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.143445 7152 ts_tablet_manager.cc:1398] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20240503 08:13:53.144724 7117 tablet_bootstrap.cc:654] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:53.145841 6503 catalog_manager.cc:5484] T 692e3e6b1f5d4e27b1bc80526338ae68 P 547814138c6346d180edab80d3927f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 547814138c6346d180edab80d3927f83 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "547814138c6346d180edab80d3927f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:53.148010 7117 tablet_bootstrap.cc:492] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83: No bootstrap required, opened a new log
I20240503 08:13:53.148167 7117 ts_tablet_manager.cc:1367] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:13:53.149483 7117 raft_consensus.cc:348] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.149718 7117 raft_consensus.cc:374] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:53.149832 7117 raft_consensus.cc:729] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547814138c6346d180edab80d3927f83, State: Initialized, Role: FOLLOWER
I20240503 08:13:53.150082 7117 consensus_queue.cc:260] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.150276 7117 raft_consensus.cc:388] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:53.150388 7117 raft_consensus.cc:482] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:53.150501 7117 raft_consensus.cc:3037] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:53.153314 7117 raft_consensus.cc:504] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.153544 7117 leader_election.cc:302] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547814138c6346d180edab80d3927f83; no voters:
I20240503 08:13:53.153745 7117 leader_election.cc:288] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:53.153801 7121 raft_consensus.cc:2781] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:53.153970 7121 raft_consensus.cc:686] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [term 1 LEADER]: Becoming Leader. State: Replica: 547814138c6346d180edab80d3927f83, State: Running, Role: LEADER
I20240503 08:13:53.154210 7117 ts_tablet_manager.cc:1398] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240503 08:13:53.154243 7121 consensus_queue.cc:237] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.156153 6503 catalog_manager.cc:5484] T 6dd0bda8e4ae4d9fb2ee058acf5479a8 P 547814138c6346d180edab80d3927f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 547814138c6346d180edab80d3927f83 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "547814138c6346d180edab80d3927f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:53.156378 6505 catalog_manager.cc:5484] T 63a7ddcb883940ea934f235e1b3d8186 P d2c9b8623c324170b7161534f65be955 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2c9b8623c324170b7161534f65be955 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "d2c9b8623c324170b7161534f65be955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:53.627871 6503 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50100:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:53.635236 6755 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1c043d0d86445c8d18c2f68c8025f4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6178c21caff34a32b0eae778020f4054]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:53.635255 6756 tablet_service.cc:1467] Processing CreateTablet for tablet 92d6ab3c0b954c15939c9a50bbbe7a69 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6178c21caff34a32b0eae778020f4054]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:53.635581 6625 tablet_service.cc:1467] Processing CreateTablet for tablet 79e408b3a1b8463c9ec1d4e2d1d24dee (DEFAULT_TABLE table=rebalance_test_table_3 [id=6178c21caff34a32b0eae778020f4054]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:53.635712 6755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1c043d0d86445c8d18c2f68c8025f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:53.635954 6756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d6ab3c0b954c15939c9a50bbbe7a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:53.635988 6625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e408b3a1b8463c9ec1d4e2d1d24dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:53.640244 7117 tablet_bootstrap.cc:492] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83: Bootstrap starting.
I20240503 08:13:53.641516 7117 tablet_bootstrap.cc:654] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:53.641882 7152 tablet_bootstrap.cc:492] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955: Bootstrap starting.
I20240503 08:13:53.643364 7152 tablet_bootstrap.cc:654] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:53.643419 7117 tablet_bootstrap.cc:492] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83: No bootstrap required, opened a new log
I20240503 08:13:53.643543 7117 ts_tablet_manager.cc:1367] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:13:53.644284 7117 raft_consensus.cc:348] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.644467 7117 raft_consensus.cc:374] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:53.644556 7117 raft_consensus.cc:729] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547814138c6346d180edab80d3927f83, State: Initialized, Role: FOLLOWER
I20240503 08:13:53.644773 7117 consensus_queue.cc:260] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.644953 7117 raft_consensus.cc:388] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:53.645030 7117 raft_consensus.cc:482] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:53.645133 7117 raft_consensus.cc:3037] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:53.647730 7152 tablet_bootstrap.cc:492] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955: No bootstrap required, opened a new log
I20240503 08:13:53.647907 7152 ts_tablet_manager.cc:1367] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:13:53.648020 7117 raft_consensus.cc:504] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.648252 7117 leader_election.cc:302] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547814138c6346d180edab80d3927f83; no voters:
I20240503 08:13:53.648447 7117 leader_election.cc:288] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:53.648504 7121 raft_consensus.cc:2781] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:53.648675 7121 raft_consensus.cc:686] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [term 1 LEADER]: Becoming Leader. State: Replica: 547814138c6346d180edab80d3927f83, State: Running, Role: LEADER
I20240503 08:13:53.648607 7152 raft_consensus.cc:348] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.648795 7152 raft_consensus.cc:374] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:53.648883 7152 raft_consensus.cc:729] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c9b8623c324170b7161534f65be955, State: Initialized, Role: FOLLOWER
I20240503 08:13:53.648916 7121 consensus_queue.cc:237] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } }
I20240503 08:13:53.649070 7117 ts_tablet_manager.cc:1398] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:13:53.649117 7152 consensus_queue.cc:260] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.649327 7152 raft_consensus.cc:388] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:53.649472 7152 raft_consensus.cc:482] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:53.649598 7152 raft_consensus.cc:3037] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:53.651139 6503 catalog_manager.cc:5484] T 79e408b3a1b8463c9ec1d4e2d1d24dee P 547814138c6346d180edab80d3927f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 547814138c6346d180edab80d3927f83 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "547814138c6346d180edab80d3927f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547814138c6346d180edab80d3927f83" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 40801 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:53.652251 7152 raft_consensus.cc:504] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.652426 7152 leader_election.cc:302] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c9b8623c324170b7161534f65be955; no voters:
I20240503 08:13:53.652611 7152 leader_election.cc:288] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:53.652678 7159 raft_consensus.cc:2781] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:53.653638 7152 ts_tablet_manager.cc:1398] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:13:53.653934 7152 tablet_bootstrap.cc:492] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955: Bootstrap starting.
I20240503 08:13:53.655925 7152 tablet_bootstrap.cc:654] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:53.657732 7152 tablet_bootstrap.cc:492] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955: No bootstrap required, opened a new log
I20240503 08:13:53.657881 7152 ts_tablet_manager.cc:1367] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:53.658597 7152 raft_consensus.cc:348] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.658766 7152 raft_consensus.cc:374] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:53.658864 7152 raft_consensus.cc:729] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c9b8623c324170b7161534f65be955, State: Initialized, Role: FOLLOWER
I20240503 08:13:53.659073 7159 raft_consensus.cc:686] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [term 1 LEADER]: Becoming Leader. State: Replica: d2c9b8623c324170b7161534f65be955, State: Running, Role: LEADER
I20240503 08:13:53.659119 7152 consensus_queue.cc:260] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.659301 7152 raft_consensus.cc:388] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:53.659399 7152 raft_consensus.cc:482] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:53.659320 7159 consensus_queue.cc:237] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.659509 7152 raft_consensus.cc:3037] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:53.661531 6505 catalog_manager.cc:5484] T 92d6ab3c0b954c15939c9a50bbbe7a69 P d2c9b8623c324170b7161534f65be955 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2c9b8623c324170b7161534f65be955 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "d2c9b8623c324170b7161534f65be955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:53.661924 7152 raft_consensus.cc:504] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.662142 7152 leader_election.cc:302] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c9b8623c324170b7161534f65be955; no voters:
I20240503 08:13:53.662322 7152 leader_election.cc:288] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:53.662547 7160 raft_consensus.cc:2781] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:53.662725 7152 ts_tablet_manager.cc:1398] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20240503 08:13:53.667466 7160 raft_consensus.cc:686] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [term 1 LEADER]: Becoming Leader. State: Replica: d2c9b8623c324170b7161534f65be955, State: Running, Role: LEADER
I20240503 08:13:53.667676 7160 consensus_queue.cc:237] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } }
I20240503 08:13:53.669654 6503 catalog_manager.cc:5484] T 0c1c043d0d86445c8d18c2f68c8025f4 P d2c9b8623c324170b7161534f65be955 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2c9b8623c324170b7161534f65be955 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "d2c9b8623c324170b7161534f65be955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c9b8623c324170b7161534f65be955" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 33949 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:54.977957 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.3:33081
--local_ip_for_outbound_sockets=127.24.123.3
--tserver_master_addrs=127.24.123.62:40289
--webserver_port=37871
--webserver_interface=127.24.123.3
--builtin_ntp_servers=127.24.123.20:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:13:55.296013 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:55.296289 7245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:55.296391 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:55.303807 7245 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:55.303982 7245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:55.304105 7245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.3
I20240503 08:13:55.313338 7245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.3:33081
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.123.3
--webserver_port=37871
--tserver_master_addrs=127.24.123.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.3
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:55.314514 7245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:55.316157 7245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:55.328459 7250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.328907 7251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.329172 7253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:55.330024 7245 server_base.cc:1034] running on GCE node
I20240503 08:13:55.330559 7245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:55.331521 7245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:55.332715 7245 hybrid_clock.cc:648] HybridClock initialized: now 1714724035332674 us; error 36 us; skew 500 ppm
I20240503 08:13:55.333014 7245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:55.335112 7245 webserver.cc:460] Webserver started at http://127.24.123.3:37871/ using document root <none> and password file <none>
I20240503 08:13:55.335813 7245 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:55.335937 7245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:55.340667 7245 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20240503 08:13:55.343142 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.344373 7245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240503 08:13:55.344527 7245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2433183a6f5046e8ba9b00190b1440a8"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:55.345274 7245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:55.363847 7245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:55.364545 7245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:55.364802 7245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:55.365635 7245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:55.367095 7245 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:55.367179 7245 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.367277 7245 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:55.367319 7245 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.394783 7245 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.3:33081
I20240503 08:13:55.394819 7371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.3:33081 every 8 connection(s)
I20240503 08:13:55.396155 7245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:13:55.402967 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 7245
I20240503 08:13:55.403577 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.4:33371
--local_ip_for_outbound_sockets=127.24.123.4
--tserver_master_addrs=127.24.123.62:40289
--webserver_port=34621
--webserver_interface=127.24.123.4
--builtin_ntp_servers=127.24.123.20:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:55.408067 7372 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40289
I20240503 08:13:55.408367 7372 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:55.409268 7372 heartbeater.cc:511] Master 127.24.123.62:40289 requested a full tablet report, sending...
I20240503 08:13:55.410777 6503 ts_manager.cc:194] Re-registered known tserver with Master: 2433183a6f5046e8ba9b00190b1440a8 (127.24.123.3:33081)
I20240503 08:13:55.411612 6503 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.3:58701
W20240503 08:13:55.723874 7376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:55.724128 7376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:55.724212 7376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:55.731459 7376 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:55.731575 7376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:55.731691 7376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.4
I20240503 08:13:55.740546 7376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.4:33371
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.123.4
--webserver_port=34621
--tserver_master_addrs=127.24.123.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.4
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:55.741732 7376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:55.743453 7376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:55.755224 7382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.755759 7384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.756203 7381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:55.757059 7376 server_base.cc:1034] running on GCE node
I20240503 08:13:55.757607 7376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:55.758486 7376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:55.759685 7376 hybrid_clock.cc:648] HybridClock initialized: now 1714724035759637 us; error 37 us; skew 500 ppm
I20240503 08:13:55.759989 7376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:55.762099 7376 webserver.cc:460] Webserver started at http://127.24.123.4:34621/ using document root <none> and password file <none>
I20240503 08:13:55.762791 7376 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:55.762936 7376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:55.767969 7376 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:55.770671 7390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.771972 7376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:13:55.772109 7376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2e330bc06a8444e983e94680338fff0d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-gkrw"
I20240503 08:13:55.773005 7376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:55.786572 7376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:55.787303 7376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:55.787560 7376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:55.788369 7376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:55.790046 7397 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 08:13:55.794798 7376 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:13:55.794917 7376 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20240503 08:13:55.795087 7376 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:13:55.797755 7376 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:13:55.797834 7376 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240503 08:13:55.798148 7397 tablet_bootstrap.cc:492] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Bootstrap starting.
I20240503 08:13:55.816577 7397 log.cc:824] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:55.821003 7397 tablet_bootstrap.cc:492] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:55.821444 7397 tablet_bootstrap.cc:492] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Bootstrap complete.
I20240503 08:13:55.822407 7397 ts_tablet_manager.cc:1367] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240503 08:13:55.830474 7397 raft_consensus.cc:348] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } }
I20240503 08:13:55.831315 7397 raft_consensus.cc:729] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e330bc06a8444e983e94680338fff0d, State: Initialized, Role: FOLLOWER
I20240503 08:13:55.832013 7397 consensus_queue.cc:260] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } }
I20240503 08:13:55.832198 7397 raft_consensus.cc:388] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:55.832368 7397 raft_consensus.cc:482] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:55.832643 7397 raft_consensus.cc:3037] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:55.837105 7376 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.4:33371
I20240503 08:13:55.837167 7504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.4:33371 every 8 connection(s)
I20240503 08:13:55.837353 7397 raft_consensus.cc:504] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } }
I20240503 08:13:55.837829 7397 leader_election.cc:302] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e330bc06a8444e983e94680338fff0d; no voters:
I20240503 08:13:55.838382 7397 leader_election.cc:288] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 08:13:55.838428 7376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:13:55.838511 7505 raft_consensus.cc:2781] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:55.840181 7397 ts_tablet_manager.cc:1398] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d: Time spent starting tablet: real 0.018s user 0.010s sys 0.008s
I20240503 08:13:55.842674 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 7376
I20240503 08:13:55.842936 7505 raft_consensus.cc:686] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [term 2 LEADER]: Becoming Leader. State: Replica: 2e330bc06a8444e983e94680338fff0d, State: Running, Role: LEADER
I20240503 08:13:55.843444 7505 consensus_queue.cc:237] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } }
I20240503 08:13:55.857327 7507 heartbeater.cc:347] Connected to a master server at 127.24.123.62:40289
I20240503 08:13:55.857599 7507 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:55.858336 7507 heartbeater.cc:511] Master 127.24.123.62:40289 requested a full tablet report, sending...
I20240503 08:13:55.860801 6503 ts_manager.cc:194] Re-registered known tserver with Master: 2e330bc06a8444e983e94680338fff0d (127.24.123.4:33371)
I20240503 08:13:55.861183 6503 catalog_manager.cc:5484] T 824c51fd02104b6589716d191b26f31a P 2e330bc06a8444e983e94680338fff0d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "2e330bc06a8444e983e94680338fff0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e330bc06a8444e983e94680338fff0d" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 33371 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:55.865965 6503 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.4:33357
I20240503 08:13:55.869123 7507 heartbeater.cc:503] Master 127.24.123.62:40289 was elected leader, sending a full tablet report...
I20240503 08:13:56.313395 7307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:56.315827 6625 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:56.329058 7439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.333931 6756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:56.414192 7372 heartbeater.cc:503] Master 127.24.123.62:40289 was elected leader, sending a full tablet report...
I20240503 08:13:56.505437 7307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:56.506333 7439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.514518 6756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:56.531086 6625 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:56.764196 7439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:56.764196 7307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:56.777992 6625 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:56.779120 6756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
df6fa78cc214403eac8aa0c3c4706f68 | 127.24.123.62:40289 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.123.20:41763 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2433183a6f5046e8ba9b00190b1440a8 | 127.24.123.3:33081 | HEALTHY | <none> | 0 | 0
2e330bc06a8444e983e94680338fff0d | 127.24.123.4:33371 | HEALTHY | <none> | 1 | 0
547814138c6346d180edab80d3927f83 | 127.24.123.1:40801 | HEALTHY | <none> | 6 | 0
d2c9b8623c324170b7161534f65be955 | 127.24.123.2:33949 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.24.123.1 | experimental | 127.24.123.1:40801
local_ip_for_outbound_sockets | 127.24.123.2 | experimental | 127.24.123.2:33949
local_ip_for_outbound_sockets | 127.24.123.3 | experimental | 127.24.123.3:33081
local_ip_for_outbound_sockets | 127.24.123.4 | experimental | 127.24.123.4:33371
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.123.1:40801
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.123.2:33949
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.123.3:33081
server_dump_info_path | /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.24.123.4:33371
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.123.20:41763 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 6
Third Quartile | 6
Maximum | 6
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:13:56.824733 25068 log_verifier.cc:126] Checking tablet 0b33dc0158fe4cc5b9966a633c2d3170
I20240503 08:13:56.831962 25068 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0b33dc0158fe4cc5b9966a633c2d3170
I20240503 08:13:56.832063 25068 log_verifier.cc:126] Checking tablet 0c1c043d0d86445c8d18c2f68c8025f4
I20240503 08:13:56.840029 25068 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 0c1c043d0d86445c8d18c2f68c8025f4
I20240503 08:13:56.840114 25068 log_verifier.cc:126] Checking tablet 1bca5ec65f5f4901947eb527a90b2244
I20240503 08:13:56.847126 25068 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 1bca5ec65f5f4901947eb527a90b2244
I20240503 08:13:56.847208 25068 log_verifier.cc:126] Checking tablet 48656f3b08464e33bdb78b47bc5a796f
I20240503 08:13:56.854763 25068 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 48656f3b08464e33bdb78b47bc5a796f
I20240503 08:13:56.854846 25068 log_verifier.cc:126] Checking tablet 63a7ddcb883940ea934f235e1b3d8186
I20240503 08:13:56.861766 25068 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 63a7ddcb883940ea934f235e1b3d8186
I20240503 08:13:56.861850 25068 log_verifier.cc:126] Checking tablet 692e3e6b1f5d4e27b1bc80526338ae68
I20240503 08:13:56.903719 25068 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 692e3e6b1f5d4e27b1bc80526338ae68
I20240503 08:13:56.903803 25068 log_verifier.cc:126] Checking tablet 6dd0bda8e4ae4d9fb2ee058acf5479a8
I20240503 08:13:56.910908 25068 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6dd0bda8e4ae4d9fb2ee058acf5479a8
I20240503 08:13:56.910988 25068 log_verifier.cc:126] Checking tablet 79e408b3a1b8463c9ec1d4e2d1d24dee
I20240503 08:13:56.918004 25068 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 79e408b3a1b8463c9ec1d4e2d1d24dee
I20240503 08:13:56.918090 25068 log_verifier.cc:126] Checking tablet 824c51fd02104b6589716d191b26f31a
I20240503 08:13:56.925112 25068 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 824c51fd02104b6589716d191b26f31a
I20240503 08:13:56.925208 25068 log_verifier.cc:126] Checking tablet 92d6ab3c0b954c15939c9a50bbbe7a69
I20240503 08:13:56.932418 25068 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 92d6ab3c0b954c15939c9a50bbbe7a69
I20240503 08:13:56.932524 25068 log_verifier.cc:126] Checking tablet 9cda1f07ae62489b82b2e8ea4e7c26ee
I20240503 08:13:56.939757 25068 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9cda1f07ae62489b82b2e8ea4e7c26ee
I20240503 08:13:56.939847 25068 log_verifier.cc:126] Checking tablet aef386c1619146d7a9da637f112899c6
I20240503 08:13:56.946924 25068 log_verifier.cc:177] Verified matching terms for 1 ops in tablet aef386c1619146d7a9da637f112899c6
I20240503 08:13:56.947014 25068 log_verifier.cc:126] Checking tablet deb7075982b544f39fafddb2cfb41c82
I20240503 08:13:56.954335 25068 log_verifier.cc:177] Verified matching terms for 2 ops in tablet deb7075982b544f39fafddb2cfb41c82
I20240503 08:13:56.955345 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 6564
I20240503 08:13:56.956420 6686 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:57.052835 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 6564
I20240503 08:13:57.068236 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 6694
I20240503 08:13:57.069211 6816 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:57.162362 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 6694
I20240503 08:13:57.177177 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 7245
I20240503 08:13:57.182912 7367 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:57.269619 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 7245
I20240503 08:13:57.283051 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 7376
I20240503 08:13:57.288410 7499 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:57.376243 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 7376
I20240503 08:13:57.390304 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 6474
I20240503 08:13:57.391333 6535 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:57.502351 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 6474
2024-05-03T08:13:57Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (11816 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (11816 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1
I20240503 08:13:57.550158 25068 test_util.cc:268] Using random seed: 1294295096
I20240503 08:13:57.551853 25068 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:13:57.551913 25068 ts_itest-base.cc:113] --------------
I20240503 08:13:57.551959 25068 ts_itest-base.cc:114] 4 tablet servers
I20240503 08:13:57.552001 25068 ts_itest-base.cc:115] 2 replicas per TS
I20240503 08:13:57.552055 25068 ts_itest-base.cc:116] --------------
2024-05-03T08:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:57Z Disabled control of system clock
I20240503 08:13:57.595108 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:44949
--webserver_interface=127.24.123.62
--webserver_port=0
--builtin_ntp_servers=127.24.123.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.123.62:44949
--location_mapping_cmd=/tmp/dist-test-taskEUWrla/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 08:13:57.911298 7628 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:13:57.911597 7628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:57.911669 7628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:57.911736 7628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:57.919206 7628 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:57.919313 7628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:57.919381 7628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:57.919426 7628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:57.919481 7628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:57.928242 7628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskEUWrla/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--ipki_ca_key_size=768
--master_addresses=127.24.123.62:44949
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:44949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.123.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:57.929481 7628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:57.931164 7628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:57.941301 7633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:57.941867 7634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:57.942258 7636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:57.942901 7628 server_base.cc:1034] running on GCE node
I20240503 08:13:57.943607 7628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:57.944909 7628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:57.946096 7628 hybrid_clock.cc:648] HybridClock initialized: now 1714724037946030 us; error 44 us; skew 500 ppm
I20240503 08:13:57.946393 7628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:57.953433 7628 webserver.cc:460] Webserver started at http://127.24.123.62:34937/ using document root <none> and password file <none>
I20240503 08:13:57.954144 7628 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:57.954267 7628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:57.954552 7628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:57.957140 7628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "028e0f66cd0e4ef8ab151daacfbb3b14"
format_stamp: "Formatted at 2024-05-03 08:13:57 on dist-test-slave-gkrw"
I20240503 08:13:57.957955 7628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "028e0f66cd0e4ef8ab151daacfbb3b14"
format_stamp: "Formatted at 2024-05-03 08:13:57 on dist-test-slave-gkrw"
I20240503 08:13:57.964746 7628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240503 08:13:57.969058 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:57.970664 7628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:57.970858 7628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
uuid: "028e0f66cd0e4ef8ab151daacfbb3b14"
format_stamp: "Formatted at 2024-05-03 08:13:57 on dist-test-slave-gkrw"
I20240503 08:13:57.971119 7628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:57.987555 7628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:57.988302 7628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:57.988564 7628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:58.012924 7628 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.62:44949
I20240503 08:13:58.012944 7693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.62:44949 every 8 connection(s)
I20240503 08:13:58.014410 7628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:13:58.018586 7694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:58.020896 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 7628
I20240503 08:13:58.021112 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:13:58.030170 7694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14: Bootstrap starting.
I20240503 08:13:58.033692 7694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:58.034808 7694 log.cc:824] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:58.037521 7694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14: No bootstrap required, opened a new log
I20240503 08:13:58.043113 7694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028e0f66cd0e4ef8ab151daacfbb3b14" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44949 } }
I20240503 08:13:58.043505 7694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:58.043613 7694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028e0f66cd0e4ef8ab151daacfbb3b14, State: Initialized, Role: FOLLOWER
I20240503 08:13:58.044174 7694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028e0f66cd0e4ef8ab151daacfbb3b14" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44949 } }
I20240503 08:13:58.044361 7694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:58.044467 7694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:58.044596 7694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:58.046550 7694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028e0f66cd0e4ef8ab151daacfbb3b14" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44949 } }
I20240503 08:13:58.047021 7694 leader_election.cc:302] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 028e0f66cd0e4ef8ab151daacfbb3b14; no voters:
I20240503 08:13:58.047544 7694 leader_election.cc:288] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:58.047654 7699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:58.048220 7699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [term 1 LEADER]: Becoming Leader. State: Replica: 028e0f66cd0e4ef8ab151daacfbb3b14, State: Running, Role: LEADER
I20240503 08:13:58.048691 7699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028e0f66cd0e4ef8ab151daacfbb3b14" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44949 } }
I20240503 08:13:58.049307 7694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:58.051024 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "028e0f66cd0e4ef8ab151daacfbb3b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028e0f66cd0e4ef8ab151daacfbb3b14" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44949 } } }
I20240503 08:13:58.051259 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:58.052625 7704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:58.052655 7701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 028e0f66cd0e4ef8ab151daacfbb3b14. Latest consensus state: current_term: 1 leader_uuid: "028e0f66cd0e4ef8ab151daacfbb3b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028e0f66cd0e4ef8ab151daacfbb3b14" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44949 } } }
I20240503 08:13:58.052901 7701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:58.056869 7704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:58.064508 7704 catalog_manager.cc:1347] Generated new cluster ID: a3fc8fb2cb924bd08cdf7b177ea14330
I20240503 08:13:58.064608 7704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:13:58.085718 7704 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:58.086853 7704 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:58.095664 7704 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 028e0f66cd0e4ef8ab151daacfbb3b14: Generated new TSK 0
I20240503 08:13:58.096675 7704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:13:58.108827 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.1:0
--local_ip_for_outbound_sockets=127.24.123.1
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44949
--builtin_ntp_servers=127.24.123.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:13:58.428547 7718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:58.428817 7718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:58.428908 7718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:58.436448 7718 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:58.436589 7718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:58.436712 7718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.1
I20240503 08:13:58.445768 7718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.1
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:58.447038 7718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:58.448675 7718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:58.460474 7723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:58.460942 7724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:58.461261 7726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:58.462185 7718 server_base.cc:1034] running on GCE node
I20240503 08:13:58.462738 7718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:58.463694 7718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:58.464931 7718 hybrid_clock.cc:648] HybridClock initialized: now 1714724038464867 us; error 56 us; skew 500 ppm
I20240503 08:13:58.465215 7718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:58.467330 7718 webserver.cc:460] Webserver started at http://127.24.123.1:42109/ using document root <none> and password file <none>
I20240503 08:13:58.468016 7718 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:58.468108 7718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:58.468451 7718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:58.470947 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "087ad327ce194194ac083345d973b083"
format_stamp: "Formatted at 2024-05-03 08:13:58 on dist-test-slave-gkrw"
I20240503 08:13:58.471650 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "087ad327ce194194ac083345d973b083"
format_stamp: "Formatted at 2024-05-03 08:13:58 on dist-test-slave-gkrw"
I20240503 08:13:58.476514 7718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 08:13:58.479714 7732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:58.480945 7718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:13:58.481072 7718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "087ad327ce194194ac083345d973b083"
format_stamp: "Formatted at 2024-05-03 08:13:58 on dist-test-slave-gkrw"
I20240503 08:13:58.481238 7718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:58.499028 7718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:58.499806 7718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:58.500061 7718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:58.500974 7718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:58.502313 7718 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:58.502404 7718 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:58.502508 7718 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:58.502575 7718 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:58.530141 7718 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.1:37315
I20240503 08:13:58.530192 7844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.1:37315 every 8 connection(s)
I20240503 08:13:58.531407 7718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:13:58.536088 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 7718
I20240503 08:13:58.536270 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:13:58.539242 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.2:0
--local_ip_for_outbound_sockets=127.24.123.2
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44949
--builtin_ntp_servers=127.24.123.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:58.543663 7845 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44949
I20240503 08:13:58.543924 7845 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:58.544674 7845 heartbeater.cc:511] Master 127.24.123.62:44949 requested a full tablet report, sending...
I20240503 08:13:58.588057 7659 ts_manager.cc:194] Registered new tserver with Master: 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:13:58.589749 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.1:43173
W20240503 08:13:58.862468 7848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:58.862735 7848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:58.862862 7848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:58.870018 7848 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:58.870126 7848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:58.870278 7848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.2
I20240503 08:13:58.879037 7848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.2
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:58.880223 7848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:58.881872 7848 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:58.894023 7855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:58.894373 7854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:58.894675 7857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:58.895646 7848 server_base.cc:1034] running on GCE node
I20240503 08:13:58.896178 7848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:58.897042 7848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:58.898265 7848 hybrid_clock.cc:648] HybridClock initialized: now 1714724038898221 us; error 40 us; skew 500 ppm
I20240503 08:13:58.898548 7848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:58.900588 7848 webserver.cc:460] Webserver started at http://127.24.123.2:46829/ using document root <none> and password file <none>
I20240503 08:13:58.901280 7848 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:58.901438 7848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:58.901743 7848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:58.904173 7848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "256fc1afb22940258b553ca2cf763fd5"
format_stamp: "Formatted at 2024-05-03 08:13:58 on dist-test-slave-gkrw"
I20240503 08:13:58.904978 7848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "256fc1afb22940258b553ca2cf763fd5"
format_stamp: "Formatted at 2024-05-03 08:13:58 on dist-test-slave-gkrw"
I20240503 08:13:58.909644 7848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:13:58.912663 7863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:58.913837 7848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:58.913983 7848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "256fc1afb22940258b553ca2cf763fd5"
format_stamp: "Formatted at 2024-05-03 08:13:58 on dist-test-slave-gkrw"
I20240503 08:13:58.914158 7848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:58.944296 7848 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:58.945104 7848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:58.945360 7848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:58.946285 7848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:58.947696 7848 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:58.947788 7848 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:58.947898 7848 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:58.947963 7848 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:58.976338 7848 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.2:41059
I20240503 08:13:58.976402 7975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.2:41059 every 8 connection(s)
I20240503 08:13:58.977569 7848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:13:58.978593 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 7848
I20240503 08:13:58.978762 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:13:58.982323 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.3:0
--local_ip_for_outbound_sockets=127.24.123.3
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44949
--builtin_ntp_servers=127.24.123.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:58.990181 7976 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44949
I20240503 08:13:58.990453 7976 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:58.991227 7976 heartbeater.cc:511] Master 127.24.123.62:44949 requested a full tablet report, sending...
I20240503 08:13:59.031891 7659 ts_manager.cc:194] Registered new tserver with Master: 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2:41059)
I20240503 08:13:59.032840 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.2:54523
W20240503 08:13:59.302783 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:59.303107 7979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:59.303238 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:59.310528 7979 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:59.310643 7979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:59.310799 7979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.3
I20240503 08:13:59.319658 7979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.3
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:59.320897 7979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:59.322494 7979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:59.334233 7985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:59.334702 7988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:59.335094 7986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:59.335757 7979 server_base.cc:1034] running on GCE node
I20240503 08:13:59.336333 7979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:59.337190 7979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:59.338382 7979 hybrid_clock.cc:648] HybridClock initialized: now 1714724039338344 us; error 29 us; skew 500 ppm
I20240503 08:13:59.338672 7979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:59.340786 7979 webserver.cc:460] Webserver started at http://127.24.123.3:45253/ using document root <none> and password file <none>
I20240503 08:13:59.341490 7979 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:59.341598 7979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:59.341887 7979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:59.344341 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e46300ca67be4f5f9d1d84ce3d14136e"
format_stamp: "Formatted at 2024-05-03 08:13:59 on dist-test-slave-gkrw"
I20240503 08:13:59.345050 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e46300ca67be4f5f9d1d84ce3d14136e"
format_stamp: "Formatted at 2024-05-03 08:13:59 on dist-test-slave-gkrw"
I20240503 08:13:59.349766 7979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240503 08:13:59.352921 7994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:59.354133 7979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:59.354277 7979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e46300ca67be4f5f9d1d84ce3d14136e"
format_stamp: "Formatted at 2024-05-03 08:13:59 on dist-test-slave-gkrw"
I20240503 08:13:59.354455 7979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:59.382000 7979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:59.382797 7979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:59.383085 7979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:59.383987 7979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:59.385409 7979 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:59.385501 7979 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:59.385605 7979 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:59.385672 7979 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:59.414055 7979 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.3:36157
I20240503 08:13:59.414110 8106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.3:36157 every 8 connection(s)
I20240503 08:13:59.415354 7979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:13:59.417524 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 7979
I20240503 08:13:59.417732 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:13:59.420846 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.4:0
--local_ip_for_outbound_sockets=127.24.123.4
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44949
--builtin_ntp_servers=127.24.123.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:59.427763 8107 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44949
I20240503 08:13:59.428081 8107 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:59.429000 8107 heartbeater.cc:511] Master 127.24.123.62:44949 requested a full tablet report, sending...
I20240503 08:13:59.471599 7659 ts_manager.cc:194] Registered new tserver with Master: e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3:36157)
I20240503 08:13:59.472934 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.3:50559
I20240503 08:13:59.592572 7845 heartbeater.cc:503] Master 127.24.123.62:44949 was elected leader, sending a full tablet report...
W20240503 08:13:59.739748 8110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:59.740017 8110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:59.740140 8110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:59.747514 8110 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:59.747642 8110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:59.747793 8110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.4
I20240503 08:13:59.756724 8110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.4
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:59.758018 8110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:59.759773 8110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:59.771047 8116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:59.771363 8119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:59.771981 8117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:59.772660 8110 server_base.cc:1034] running on GCE node
I20240503 08:13:59.773172 8110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:59.774039 8110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:59.775210 8110 hybrid_clock.cc:648] HybridClock initialized: now 1714724039775155 us; error 49 us; skew 500 ppm
I20240503 08:13:59.775496 8110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:59.777551 8110 webserver.cc:460] Webserver started at http://127.24.123.4:44183/ using document root <none> and password file <none>
I20240503 08:13:59.778240 8110 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:59.778340 8110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:59.778626 8110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:59.781071 8110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "34a09bb831e64f278fd57493bf6554d1"
format_stamp: "Formatted at 2024-05-03 08:13:59 on dist-test-slave-gkrw"
I20240503 08:13:59.781773 8110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "34a09bb831e64f278fd57493bf6554d1"
format_stamp: "Formatted at 2024-05-03 08:13:59 on dist-test-slave-gkrw"
I20240503 08:13:59.786520 8110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240503 08:13:59.789537 8125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:59.790761 8110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:59.790925 8110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "34a09bb831e64f278fd57493bf6554d1"
format_stamp: "Formatted at 2024-05-03 08:13:59 on dist-test-slave-gkrw"
I20240503 08:13:59.791095 8110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:59.817950 8110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:59.818737 8110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:59.819033 8110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:59.819938 8110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:59.821308 8110 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:59.821389 8110 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:59.821471 8110 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:59.821521 8110 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:59.849099 8110 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.4:43277
I20240503 08:13:59.849184 8237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.4:43277 every 8 connection(s)
I20240503 08:13:59.850415 8110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:13:59.859537 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 8110
I20240503 08:13:59.859730 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:13:59.862565 8238 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44949
I20240503 08:13:59.862821 8238 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:59.863682 8238 heartbeater.cc:511] Master 127.24.123.62:44949 requested a full tablet report, sending...
I20240503 08:13:59.904716 7659 ts_manager.cc:194] Registered new tserver with Master: 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:13:59.905582 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.4:35395
I20240503 08:13:59.911952 25068 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:13:59.926718 25068 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:13:59.926899 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 7848
I20240503 08:13:59.934099 7971 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:00.011349 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 7848
I20240503 08:14:00.025177 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 7979
I20240503 08:14:00.032763 8102 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:00.110819 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 7979
I20240503 08:14:00.908344 8238 heartbeater.cc:503] Master 127.24.123.62:44949 was elected leader, sending a full tablet report...
I20240503 08:14:03.875697 7659 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33132:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:03.882686 7659 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 08:14:03.909355 8173 tablet_service.cc:1467] Processing CreateTablet for tablet fec10fdaedf44a46abc4a3f970a4305c (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef773cd915eb4ae7bd04beb5c55ca9af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:03.909600 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f0c69966b4703b025e8f915d59c82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef773cd915eb4ae7bd04beb5c55ca9af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:03.909334 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f0c69966b4703b025e8f915d59c82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef773cd915eb4ae7bd04beb5c55ca9af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:03.909361 7780 tablet_service.cc:1467] Processing CreateTablet for tablet fec10fdaedf44a46abc4a3f970a4305c (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef773cd915eb4ae7bd04beb5c55ca9af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:03.910396 8172 tablet_service.cc:1467] Processing CreateTablet for tablet d761e683bf8b4e3f853f42ae38a2a939 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef773cd915eb4ae7bd04beb5c55ca9af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:03.909327 7779 tablet_service.cc:1467] Processing CreateTablet for tablet d761e683bf8b4e3f853f42ae38a2a939 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef773cd915eb4ae7bd04beb5c55ca9af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:03.910851 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3f0c69966b4703b025e8f915d59c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:03.910851 7780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec10fdaedf44a46abc4a3f970a4305c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:03.911336 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d761e683bf8b4e3f853f42ae38a2a939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:03.911538 7779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d761e683bf8b4e3f853f42ae38a2a939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:03.912781 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3f0c69966b4703b025e8f915d59c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:03.913017 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec10fdaedf44a46abc4a3f970a4305c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:03.923422 8260 tablet_bootstrap.cc:492] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083: Bootstrap starting.
I20240503 08:14:03.926748 8260 tablet_bootstrap.cc:654] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:03.927788 8261 tablet_bootstrap.cc:492] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1: Bootstrap starting.
I20240503 08:14:03.927951 8260 log.cc:824] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:03.930719 8260 tablet_bootstrap.cc:492] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083: No bootstrap required, opened a new log
I20240503 08:14:03.931020 8260 ts_tablet_manager.cc:1367] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:14:03.931315 8261 tablet_bootstrap.cc:654] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:03.932305 8261 log.cc:824] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:03.934582 8261 tablet_bootstrap.cc:492] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1: No bootstrap required, opened a new log
I20240503 08:14:03.934798 8261 ts_tablet_manager.cc:1367] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:14:03.937203 8260 raft_consensus.cc:348] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.937503 8260 raft_consensus.cc:374] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:03.937603 8260 raft_consensus.cc:729] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Initialized, Role: FOLLOWER
I20240503 08:14:03.938139 8260 consensus_queue.cc:260] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.939496 8260 ts_tablet_manager.cc:1398] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 08:14:03.939910 8260 tablet_bootstrap.cc:492] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083: Bootstrap starting.
I20240503 08:14:03.940850 8261 raft_consensus.cc:348] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.941135 8261 raft_consensus.cc:374] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:03.941228 8261 raft_consensus.cc:729] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:03.941274 8260 tablet_bootstrap.cc:654] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:03.941756 8261 consensus_queue.cc:260] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.943167 8260 tablet_bootstrap.cc:492] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083: No bootstrap required, opened a new log
I20240503 08:14:03.943347 8260 ts_tablet_manager.cc:1367] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:14:03.943863 8261 ts_tablet_manager.cc:1398] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20240503 08:14:03.944264 8261 tablet_bootstrap.cc:492] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1: Bootstrap starting.
I20240503 08:14:03.944339 8260 raft_consensus.cc:348] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.944536 8260 raft_consensus.cc:374] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:03.944667 8260 raft_consensus.cc:729] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Initialized, Role: FOLLOWER
I20240503 08:14:03.944904 8260 consensus_queue.cc:260] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.945375 8260 ts_tablet_manager.cc:1398] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:03.945649 8260 tablet_bootstrap.cc:492] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083: Bootstrap starting.
I20240503 08:14:03.945976 8261 tablet_bootstrap.cc:654] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:03.947149 8260 tablet_bootstrap.cc:654] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:03.951025 8266 raft_consensus.cc:482] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:03.951331 8266 raft_consensus.cc:504] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.952018 8261 tablet_bootstrap.cc:492] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1: No bootstrap required, opened a new log
I20240503 08:14:03.952140 8261 ts_tablet_manager.cc:1367] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240503 08:14:03.952178 8260 tablet_bootstrap.cc:492] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083: No bootstrap required, opened a new log
I20240503 08:14:03.952380 8260 ts_tablet_manager.cc:1367] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:14:03.952919 8261 raft_consensus.cc:348] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.953012 8260 raft_consensus.cc:348] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.953244 8260 raft_consensus.cc:374] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:03.953249 8261 raft_consensus.cc:374] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:03.953356 8261 raft_consensus.cc:729] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:03.952963 8266 leader_election.cc:288] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:03.953526 8260 raft_consensus.cc:729] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Initialized, Role: FOLLOWER
I20240503 08:14:03.953655 8261 consensus_queue.cc:260] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.953784 8260 consensus_queue.cc:260] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.954121 8261 ts_tablet_manager.cc:1398] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:03.954259 8260 ts_tablet_manager.cc:1398] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:03.954743 8261 tablet_bootstrap.cc:492] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: Bootstrap starting.
I20240503 08:14:03.956357 8261 tablet_bootstrap.cc:654] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:03.959565 8261 tablet_bootstrap.cc:492] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: No bootstrap required, opened a new log
I20240503 08:14:03.959709 8261 ts_tablet_manager.cc:1367] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 08:14:03.960225 8261 raft_consensus.cc:348] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.960386 8261 raft_consensus.cc:374] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:03.960459 8261 raft_consensus.cc:729] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:03.960675 8261 consensus_queue.cc:260] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.961303 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761e683bf8b4e3f853f42ae38a2a939" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087ad327ce194194ac083345d973b083" is_pre_election: true
I20240503 08:14:03.961771 7800 raft_consensus.cc:2454] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 0.
I20240503 08:14:03.962393 8261 ts_tablet_manager.cc:1398] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:14:03.962338 8128 leader_election.cc:302] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:03.962693 8266 raft_consensus.cc:2781] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:03.962829 8266 raft_consensus.cc:482] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:03.962962 8266 raft_consensus.cc:3037] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:03.965262 8266 raft_consensus.cc:504] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.965667 8266 leader_election.cc:288] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 election: Requested vote from peers 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:03.966060 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761e683bf8b4e3f853f42ae38a2a939" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087ad327ce194194ac083345d973b083"
I20240503 08:14:03.966198 7800 raft_consensus.cc:3037] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:03.968102 7800 raft_consensus.cc:2454] T d761e683bf8b4e3f853f42ae38a2a939 P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 1.
I20240503 08:14:03.968401 8128 leader_election.cc:302] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:03.968642 8266 raft_consensus.cc:2781] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:03.969023 8266 raft_consensus.cc:686] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Becoming Leader. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Running, Role: LEADER
I20240503 08:14:03.969390 8266 consensus_queue.cc:237] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:03.973389 7658 catalog_manager.cc:5484] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } }
W20240503 08:14:04.036280 7846 tablet.cc:2369] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:14:04.104513 8239 tablet.cc:2369] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:04.220849 8264 raft_consensus.cc:482] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:04.221083 8264 raft_consensus.cc:504] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:04.222499 8264 leader_election.cc:288] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:04.228581 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f0c69966b4703b025e8f915d59c82" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1" is_pre_election: true
I20240503 08:14:04.228953 8193 raft_consensus.cc:2454] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 0.
I20240503 08:14:04.229485 7735 leader_election.cc:302] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:04.229843 8264 raft_consensus.cc:2781] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:04.229961 8264 raft_consensus.cc:482] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:04.230038 8264 raft_consensus.cc:3037] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:04.231990 8264 raft_consensus.cc:504] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:04.232455 8264 leader_election.cc:288] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Requested vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:04.233006 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f0c69966b4703b025e8f915d59c82" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1"
I20240503 08:14:04.233163 8193 raft_consensus.cc:3037] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:04.235047 8193 raft_consensus.cc:2454] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 1.
I20240503 08:14:04.235424 7735 leader_election.cc:302] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:04.235683 8264 raft_consensus.cc:2781] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:04.236090 8264 raft_consensus.cc:686] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Becoming Leader. State: Replica: 087ad327ce194194ac083345d973b083, State: Running, Role: LEADER
I20240503 08:14:04.236548 8264 consensus_queue.cc:237] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:04.240535 7658 catalog_manager.cc:5484] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 087ad327ce194194ac083345d973b083 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:04.261656 8266 raft_consensus.cc:482] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:04.261857 8266 raft_consensus.cc:504] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:04.262318 8266 leader_election.cc:288] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:04.262902 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec10fdaedf44a46abc4a3f970a4305c" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087ad327ce194194ac083345d973b083" is_pre_election: true
I20240503 08:14:04.263168 7800 raft_consensus.cc:2454] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 0.
I20240503 08:14:04.263605 8128 leader_election.cc:302] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:04.263904 8266 raft_consensus.cc:2781] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:04.264005 8266 raft_consensus.cc:482] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:04.264091 8266 raft_consensus.cc:3037] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:04.266037 8266 raft_consensus.cc:504] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:04.266513 8266 leader_election.cc:288] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 election: Requested vote from peers 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:04.266979 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec10fdaedf44a46abc4a3f970a4305c" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087ad327ce194194ac083345d973b083"
I20240503 08:14:04.267179 7800 raft_consensus.cc:3037] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:04.269022 7800 raft_consensus.cc:2454] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 1.
I20240503 08:14:04.269382 8128 leader_election.cc:302] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:04.269682 8266 raft_consensus.cc:2781] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:04.269824 8266 raft_consensus.cc:686] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Becoming Leader. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Running, Role: LEADER
I20240503 08:14:04.270051 8266 consensus_queue.cc:237] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:04.272348 7658 catalog_manager.cc:5484] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:04.581159 8266 consensus_queue.cc:1035] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:04.672698 8277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:04.673017 8277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:04.680276 8277 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:14:04.806330 7800 raft_consensus.cc:1261] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:04.807834 8266 consensus_queue.cc:1035] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:04.819541 8193 raft_consensus.cc:1261] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:04.820438 8264 consensus_queue.cc:1035] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:04.849632 8312 mvcc.cc:205] Tried to move back new op lower bound from 7023509687572627456 to 7023509685195923456. Current Snapshot: MvccSnapshot[applied={T|T < 7023509687521808384}]
I20240503 08:14:04.864593 8279 mvcc.cc:205] Tried to move back new op lower bound from 7023509687572627456 to 7023509685195923456. Current Snapshot: MvccSnapshot[applied={T|T < 7023509687521808384}]
INSERT report
rows total: 2000
time total: 167.912 ms
time per row: 0.0839559 ms
I20240503 08:14:04.985364 7656 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33132:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:04.986133 7656 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 08:14:04.993798 7779 tablet_service.cc:1467] Processing CreateTablet for tablet 4feffc41ba284c7787642d3497c19a4e (DEFAULT_TABLE table=rebalance_test_table_1 [id=beeccc26a3874634b97dd0b996caa598]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:04.993800 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 4feffc41ba284c7787642d3497c19a4e (DEFAULT_TABLE table=rebalance_test_table_1 [id=beeccc26a3874634b97dd0b996caa598]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:04.994259 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4feffc41ba284c7787642d3497c19a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.994266 7779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4feffc41ba284c7787642d3497c19a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.994340 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 90506a0036834a3180d6eaa0d030c1ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=beeccc26a3874634b97dd0b996caa598]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:04.994426 8172 tablet_service.cc:1467] Processing CreateTablet for tablet 90506a0036834a3180d6eaa0d030c1ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=beeccc26a3874634b97dd0b996caa598]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:04.994740 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90506a0036834a3180d6eaa0d030c1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:04.994994 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 987da8ea2afa4f098e101a2e8c3ba53d (DEFAULT_TABLE table=rebalance_test_table_1 [id=beeccc26a3874634b97dd0b996caa598]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:04.995333 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 987da8ea2afa4f098e101a2e8c3ba53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:05.002009 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90506a0036834a3180d6eaa0d030c1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:05.002404 7780 tablet_service.cc:1467] Processing CreateTablet for tablet 987da8ea2afa4f098e101a2e8c3ba53d (DEFAULT_TABLE table=rebalance_test_table_1 [id=beeccc26a3874634b97dd0b996caa598]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:05.002789 7780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 987da8ea2afa4f098e101a2e8c3ba53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:05.005110 8330 tablet_bootstrap.cc:492] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1: Bootstrap starting.
I20240503 08:14:05.006744 8330 tablet_bootstrap.cc:654] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:05.009696 8329 tablet_bootstrap.cc:492] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083: Bootstrap starting.
I20240503 08:14:05.010742 8330 tablet_bootstrap.cc:492] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1: No bootstrap required, opened a new log
I20240503 08:14:05.010911 8330 ts_tablet_manager.cc:1367] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:14:05.011348 8329 tablet_bootstrap.cc:654] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:05.011699 8330 raft_consensus.cc:348] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.011906 8330 raft_consensus.cc:374] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:05.011999 8330 raft_consensus.cc:729] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:05.012223 8330 consensus_queue.cc:260] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.012866 8330 ts_tablet_manager.cc:1398] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:05.013200 8330 tablet_bootstrap.cc:492] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1: Bootstrap starting.
I20240503 08:14:05.014328 8330 tablet_bootstrap.cc:654] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:05.016022 8330 tablet_bootstrap.cc:492] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1: No bootstrap required, opened a new log
I20240503 08:14:05.016224 8330 ts_tablet_manager.cc:1367] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:14:05.016994 8330 raft_consensus.cc:348] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.017189 8330 raft_consensus.cc:374] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:05.017275 8330 raft_consensus.cc:729] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:05.017479 8330 consensus_queue.cc:260] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.018010 8330 ts_tablet_manager.cc:1398] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:05.018173 8329 tablet_bootstrap.cc:492] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083: No bootstrap required, opened a new log
I20240503 08:14:05.018297 8330 tablet_bootstrap.cc:492] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1: Bootstrap starting.
I20240503 08:14:05.018350 8329 ts_tablet_manager.cc:1367] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:14:05.019273 8329 raft_consensus.cc:348] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.019522 8329 raft_consensus.cc:374] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:05.019660 8329 raft_consensus.cc:729] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Initialized, Role: FOLLOWER
I20240503 08:14:05.019822 8330 tablet_bootstrap.cc:654] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:05.019987 8329 consensus_queue.cc:260] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.020740 8329 ts_tablet_manager.cc:1398] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:05.021548 8330 tablet_bootstrap.cc:492] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1: No bootstrap required, opened a new log
I20240503 08:14:05.021687 8330 ts_tablet_manager.cc:1367] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:05.022339 8330 raft_consensus.cc:348] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.022548 8330 raft_consensus.cc:374] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:05.022702 8330 raft_consensus.cc:729] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:05.023025 8329 tablet_bootstrap.cc:492] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083: Bootstrap starting.
I20240503 08:14:05.022989 8330 consensus_queue.cc:260] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.023511 8330 ts_tablet_manager.cc:1398] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:05.025653 8329 tablet_bootstrap.cc:654] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:05.027318 8329 tablet_bootstrap.cc:492] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083: No bootstrap required, opened a new log
I20240503 08:14:05.027508 8329 ts_tablet_manager.cc:1367] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:14:05.028064 8329 raft_consensus.cc:348] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.028205 8329 raft_consensus.cc:374] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:05.028271 8329 raft_consensus.cc:729] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Initialized, Role: FOLLOWER
I20240503 08:14:05.028463 8329 consensus_queue.cc:260] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.028875 8329 ts_tablet_manager.cc:1398] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:14:05.029208 8329 tablet_bootstrap.cc:492] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083: Bootstrap starting.
I20240503 08:14:05.030331 8329 tablet_bootstrap.cc:654] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:05.032172 8329 tablet_bootstrap.cc:492] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083: No bootstrap required, opened a new log
I20240503 08:14:05.032342 8329 ts_tablet_manager.cc:1367] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:14:05.032960 8329 raft_consensus.cc:348] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.033138 8329 raft_consensus.cc:374] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:05.033231 8329 raft_consensus.cc:729] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Initialized, Role: FOLLOWER
I20240503 08:14:05.033434 8329 consensus_queue.cc:260] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.033854 8329 ts_tablet_manager.cc:1398] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:14:05.036156 8266 raft_consensus.cc:482] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:05.036310 8266 raft_consensus.cc:504] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.036852 8266 leader_election.cc:288] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:05.037233 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4feffc41ba284c7787642d3497c19a4e" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087ad327ce194194ac083345d973b083" is_pre_election: true
I20240503 08:14:05.037415 7800 raft_consensus.cc:2454] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 0.
I20240503 08:14:05.037786 8128 leader_election.cc:302] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:05.038064 8266 raft_consensus.cc:2781] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:05.038190 8266 raft_consensus.cc:482] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:05.038290 8266 raft_consensus.cc:3037] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:05.040951 8266 raft_consensus.cc:504] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.041393 8266 leader_election.cc:288] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 election: Requested vote from peers 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:05.041806 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4feffc41ba284c7787642d3497c19a4e" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087ad327ce194194ac083345d973b083"
I20240503 08:14:05.041982 7800 raft_consensus.cc:3037] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:05.043746 7800 raft_consensus.cc:2454] T 4feffc41ba284c7787642d3497c19a4e P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 1.
I20240503 08:14:05.044080 8128 leader_election.cc:302] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:05.044348 8266 raft_consensus.cc:2781] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:05.044489 8266 raft_consensus.cc:686] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Becoming Leader. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Running, Role: LEADER
I20240503 08:14:05.044751 8266 consensus_queue.cc:237] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.046973 7656 catalog_manager.cc:5484] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:05.113727 8266 raft_consensus.cc:482] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:05.113937 8266 raft_consensus.cc:504] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.114400 8266 leader_election.cc:288] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:05.115023 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90506a0036834a3180d6eaa0d030c1ec" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087ad327ce194194ac083345d973b083" is_pre_election: true
I20240503 08:14:05.115221 7800 raft_consensus.cc:2454] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 0.
I20240503 08:14:05.115819 8128 leader_election.cc:302] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:05.116113 8266 raft_consensus.cc:2781] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:05.116218 8266 raft_consensus.cc:482] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:05.116334 8266 raft_consensus.cc:3037] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:05.118197 8266 raft_consensus.cc:504] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.118654 8266 leader_election.cc:288] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 election: Requested vote from peers 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:05.119143 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90506a0036834a3180d6eaa0d030c1ec" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087ad327ce194194ac083345d973b083"
I20240503 08:14:05.119316 7800 raft_consensus.cc:3037] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:05.121140 7800 raft_consensus.cc:2454] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 1.
I20240503 08:14:05.121505 8128 leader_election.cc:302] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:05.121781 8266 raft_consensus.cc:2781] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:05.121963 8266 raft_consensus.cc:686] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Becoming Leader. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Running, Role: LEADER
I20240503 08:14:05.122179 8266 consensus_queue.cc:237] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.124477 7656 catalog_manager.cc:5484] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:05.367623 8264 raft_consensus.cc:482] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:05.367797 8264 raft_consensus.cc:504] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.368321 8264 leader_election.cc:288] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:05.368979 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "987da8ea2afa4f098e101a2e8c3ba53d" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1" is_pre_election: true
I20240503 08:14:05.369192 8193 raft_consensus.cc:2454] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 0.
I20240503 08:14:05.369625 7735 leader_election.cc:302] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:05.369891 8264 raft_consensus.cc:2781] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:05.369994 8264 raft_consensus.cc:482] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:05.370074 8264 raft_consensus.cc:3037] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:05.371793 8264 raft_consensus.cc:504] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.372184 8264 leader_election.cc:288] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Requested vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:05.372656 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "987da8ea2afa4f098e101a2e8c3ba53d" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1"
I20240503 08:14:05.372807 8193 raft_consensus.cc:3037] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:05.374754 8193 raft_consensus.cc:2454] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 1.
I20240503 08:14:05.375100 7735 leader_election.cc:302] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:05.375348 8264 raft_consensus.cc:2781] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:05.375468 8264 raft_consensus.cc:686] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Becoming Leader. State: Replica: 087ad327ce194194ac083345d973b083, State: Running, Role: LEADER
I20240503 08:14:05.375705 8264 consensus_queue.cc:237] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:05.377816 7656 catalog_manager.cc:5484] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 087ad327ce194194ac083345d973b083 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:05.520223 8266 consensus_queue.cc:1035] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:05.534293 8271 consensus_queue.cc:1035] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:05.756547 8264 consensus_queue.cc:1035] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:05.798553 8340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:05.798957 8340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:05.806550 8340 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 160.997 ms
time per row: 0.0804987 ms
I20240503 08:14:06.104379 7656 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33132:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:06.105099 7656 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 08:14:06.112540 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 746748dba86242da9a54e1346ccfa2c9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26b2dfa77389481e8c16612262154ee6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:06.112831 8172 tablet_service.cc:1467] Processing CreateTablet for tablet 746748dba86242da9a54e1346ccfa2c9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26b2dfa77389481e8c16612262154ee6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:06.112990 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746748dba86242da9a54e1346ccfa2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:06.113016 8173 tablet_service.cc:1467] Processing CreateTablet for tablet f6265fadd82a446d9b04efd7e85638a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26b2dfa77389481e8c16612262154ee6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:06.113206 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746748dba86242da9a54e1346ccfa2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:06.113451 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6265fadd82a446d9b04efd7e85638a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:06.113921 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 6e10ce115a9d4402bf4db426db4fbeac (DEFAULT_TABLE table=rebalance_test_table_2 [id=26b2dfa77389481e8c16612262154ee6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:06.114298 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e10ce115a9d4402bf4db426db4fbeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:06.117923 7779 tablet_service.cc:1467] Processing CreateTablet for tablet f6265fadd82a446d9b04efd7e85638a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=26b2dfa77389481e8c16612262154ee6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:06.118028 7780 tablet_service.cc:1467] Processing CreateTablet for tablet 6e10ce115a9d4402bf4db426db4fbeac (DEFAULT_TABLE table=rebalance_test_table_2 [id=26b2dfa77389481e8c16612262154ee6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:06.118245 7779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6265fadd82a446d9b04efd7e85638a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:06.118436 7780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e10ce115a9d4402bf4db426db4fbeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:06.125036 8390 tablet_bootstrap.cc:492] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1: Bootstrap starting.
I20240503 08:14:06.125337 8391 tablet_bootstrap.cc:492] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083: Bootstrap starting.
I20240503 08:14:06.126652 8390 tablet_bootstrap.cc:654] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:06.127018 8391 tablet_bootstrap.cc:654] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:06.128268 8390 tablet_bootstrap.cc:492] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1: No bootstrap required, opened a new log
I20240503 08:14:06.128408 8390 ts_tablet_manager.cc:1367] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:14:06.128716 8391 tablet_bootstrap.cc:492] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083: No bootstrap required, opened a new log
I20240503 08:14:06.128847 8391 ts_tablet_manager.cc:1367] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:14:06.129206 8390 raft_consensus.cc:348] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:06.129406 8390 raft_consensus.cc:374] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:06.129505 8390 raft_consensus.cc:729] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:06.129508 8391 raft_consensus.cc:348] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.129690 8391 raft_consensus.cc:374] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:06.129781 8391 raft_consensus.cc:729] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Initialized, Role: FOLLOWER
I20240503 08:14:06.129726 8390 consensus_queue.cc:260] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:06.129981 8391 consensus_queue.cc:260] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.130257 8390 ts_tablet_manager.cc:1398] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:06.130507 8390 tablet_bootstrap.cc:492] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1: Bootstrap starting.
I20240503 08:14:06.130585 8391 ts_tablet_manager.cc:1398] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:06.130820 8391 tablet_bootstrap.cc:492] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: Bootstrap starting.
I20240503 08:14:06.132056 8390 tablet_bootstrap.cc:654] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:06.132177 8391 tablet_bootstrap.cc:654] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:06.134938 8391 tablet_bootstrap.cc:492] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: No bootstrap required, opened a new log
I20240503 08:14:06.135080 8391 ts_tablet_manager.cc:1367] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:14:06.135843 8391 raft_consensus.cc:348] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.136044 8391 raft_consensus.cc:374] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:06.136102 8391 raft_consensus.cc:729] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Initialized, Role: FOLLOWER
I20240503 08:14:06.136301 8391 consensus_queue.cc:260] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.136809 8390 tablet_bootstrap.cc:492] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1: No bootstrap required, opened a new log
I20240503 08:14:06.136831 8391 ts_tablet_manager.cc:1398] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:06.136971 8390 ts_tablet_manager.cc:1367] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:14:06.137061 8391 tablet_bootstrap.cc:492] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083: Bootstrap starting.
I20240503 08:14:06.137647 8390 raft_consensus.cc:348] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.137851 8390 raft_consensus.cc:374] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:06.137933 8390 raft_consensus.cc:729] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:06.138141 8390 consensus_queue.cc:260] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.138490 8391 tablet_bootstrap.cc:654] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:06.138579 8390 ts_tablet_manager.cc:1398] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:06.138897 8390 tablet_bootstrap.cc:492] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1: Bootstrap starting.
I20240503 08:14:06.140398 8390 tablet_bootstrap.cc:654] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:06.140827 8324 raft_consensus.cc:482] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:06.140978 8324 raft_consensus.cc:504] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.141597 8324 leader_election.cc:288] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:06.142135 8390 tablet_bootstrap.cc:492] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1: No bootstrap required, opened a new log
I20240503 08:14:06.142078 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e10ce115a9d4402bf4db426db4fbeac" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1" is_pre_election: true
I20240503 08:14:06.142257 8390 ts_tablet_manager.cc:1367] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:06.142284 8193 raft_consensus.cc:2454] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 0.
I20240503 08:14:06.142655 7735 leader_election.cc:302] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:06.142971 8324 raft_consensus.cc:2781] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:06.142927 8390 raft_consensus.cc:348] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.143074 8324 raft_consensus.cc:482] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:06.143085 8390 raft_consensus.cc:374] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:06.143155 8324 raft_consensus.cc:3037] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:06.143169 8390 raft_consensus.cc:729] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:06.143357 8390 consensus_queue.cc:260] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.143743 8390 ts_tablet_manager.cc:1398] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 08:14:06.144603 8391 tablet_bootstrap.cc:492] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083: No bootstrap required, opened a new log
I20240503 08:14:06.144721 8391 ts_tablet_manager.cc:1367] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:14:06.145098 8324 raft_consensus.cc:504] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.145265 8391 raft_consensus.cc:348] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:06.145409 8391 raft_consensus.cc:374] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:06.145509 8324 leader_election.cc:288] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Requested vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:06.145509 8391 raft_consensus.cc:729] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Initialized, Role: FOLLOWER
I20240503 08:14:06.145769 8391 consensus_queue.cc:260] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:06.145999 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e10ce115a9d4402bf4db426db4fbeac" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1"
I20240503 08:14:06.146186 8193 raft_consensus.cc:3037] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:06.146332 8391 ts_tablet_manager.cc:1398] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:06.147545 8324 raft_consensus.cc:482] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:06.147672 8324 raft_consensus.cc:504] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:06.148099 8324 leader_election.cc:288] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:06.148550 8193 raft_consensus.cc:2454] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 1.
I20240503 08:14:06.148844 7735 leader_election.cc:302] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:06.148885 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6265fadd82a446d9b04efd7e85638a7" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1" is_pre_election: true
I20240503 08:14:06.149072 8324 raft_consensus.cc:2781] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:06.149152 8192 raft_consensus.cc:2454] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 0.
I20240503 08:14:06.149209 8324 raft_consensus.cc:686] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Becoming Leader. State: Replica: 087ad327ce194194ac083345d973b083, State: Running, Role: LEADER
I20240503 08:14:06.149533 7735 leader_election.cc:302] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:06.149464 8324 consensus_queue.cc:237] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.149778 8264 raft_consensus.cc:2781] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:06.149865 8264 raft_consensus.cc:482] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:06.149987 8264 raft_consensus.cc:3037] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:06.152112 8264 raft_consensus.cc:504] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:06.152570 8264 leader_election.cc:288] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Requested vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:06.152958 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6265fadd82a446d9b04efd7e85638a7" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1"
I20240503 08:14:06.153095 8192 raft_consensus.cc:3037] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:06.153865 7650 catalog_manager.cc:5484] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 087ad327ce194194ac083345d973b083 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:06.155468 8192 raft_consensus.cc:2454] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 1.
I20240503 08:14:06.155792 7735 leader_election.cc:302] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:06.156013 8264 raft_consensus.cc:2781] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:06.156181 8264 raft_consensus.cc:686] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Becoming Leader. State: Replica: 087ad327ce194194ac083345d973b083, State: Running, Role: LEADER
I20240503 08:14:06.156406 8264 consensus_queue.cc:237] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } }
I20240503 08:14:06.158450 7650 catalog_manager.cc:5484] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 087ad327ce194194ac083345d973b083 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:06.195334 8264 raft_consensus.cc:482] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:06.195542 8264 raft_consensus.cc:504] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.196049 8264 leader_election.cc:288] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:06.196588 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746748dba86242da9a54e1346ccfa2c9" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1" is_pre_election: true
I20240503 08:14:06.196810 8192 raft_consensus.cc:2454] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 0.
I20240503 08:14:06.197203 7735 leader_election.cc:302] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:06.197455 8264 raft_consensus.cc:2781] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:06.197553 8264 raft_consensus.cc:482] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:06.197633 8264 raft_consensus.cc:3037] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:06.199472 8264 raft_consensus.cc:504] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.199934 8264 leader_election.cc:288] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Requested vote from peers 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:06.200512 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746748dba86242da9a54e1346ccfa2c9" candidate_uuid: "087ad327ce194194ac083345d973b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a09bb831e64f278fd57493bf6554d1"
I20240503 08:14:06.200704 8192 raft_consensus.cc:3037] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:06.202605 8192 raft_consensus.cc:2454] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 087ad327ce194194ac083345d973b083 in term 1.
I20240503 08:14:06.203011 7735 leader_election.cc:302] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 087ad327ce194194ac083345d973b083, 34a09bb831e64f278fd57493bf6554d1; no voters:
I20240503 08:14:06.203320 8264 raft_consensus.cc:2781] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:06.203438 8264 raft_consensus.cc:686] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Becoming Leader. State: Replica: 087ad327ce194194ac083345d973b083, State: Running, Role: LEADER
I20240503 08:14:06.203667 8264 consensus_queue.cc:237] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } }
I20240503 08:14:06.206009 7650 catalog_manager.cc:5484] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 087ad327ce194194ac083345d973b083 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: UNKNOWN } } }
W20240503 08:14:06.545637 8401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:06.545961 8401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:06.553694 8401 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:14:06.671272 8192 raft_consensus.cc:1261] T f6265fadd82a446d9b04efd7e85638a7 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:06.671890 8264 consensus_queue.cc:1035] T f6265fadd82a446d9b04efd7e85638a7 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:06.676821 8193 raft_consensus.cc:1261] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:06.677371 8264 consensus_queue.cc:1035] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:06.683277 8191 raft_consensus.cc:1261] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:06.683933 8264 consensus_queue.cc:1035] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 159.627 ms
time per row: 0.0798137 ms
I20240503 08:14:06.844781 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.2:41059
--local_ip_for_outbound_sockets=127.24.123.2
--tserver_master_addrs=127.24.123.62:44949
--webserver_port=46829
--webserver_interface=127.24.123.2
--builtin_ntp_servers=127.24.123.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:14:07.182548 8448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:07.182817 8448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:07.182960 8448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:07.190378 8448 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:07.190497 8448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:07.190642 8448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.2
I20240503 08:14:07.199712 8448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.2:41059
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.123.2
--webserver_port=46829
--tserver_master_addrs=127.24.123.62:44949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.2
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:07.201026 8448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:07.202726 8448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:07.214543 8454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:07.215003 8456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:07.215310 8453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:07.216104 8448 server_base.cc:1034] running on GCE node
I20240503 08:14:07.216636 8448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:07.217471 8448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:07.218683 8448 hybrid_clock.cc:648] HybridClock initialized: now 1714724047218631 us; error 39 us; skew 500 ppm
I20240503 08:14:07.219028 8448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:07.221256 8448 webserver.cc:460] Webserver started at http://127.24.123.2:46829/ using document root <none> and password file <none>
I20240503 08:14:07.221953 8448 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:07.222074 8448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:07.227180 8448 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:07.229696 8462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:07.230968 8448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:07.231117 8448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "256fc1afb22940258b553ca2cf763fd5"
format_stamp: "Formatted at 2024-05-03 08:13:58 on dist-test-slave-gkrw"
I20240503 08:14:07.231858 8448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:07.250408 8448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:07.251250 8448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:07.251547 8448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:07.252486 8448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:07.253983 8448 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:07.254052 8448 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:07.254132 8448 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:07.254173 8448 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:07.282485 8448 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.2:41059
I20240503 08:14:07.282537 8574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.2:41059 every 8 connection(s)
I20240503 08:14:07.283946 8448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:14:07.287253 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 8448
I20240503 08:14:07.287837 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.3:36157
--local_ip_for_outbound_sockets=127.24.123.3
--tserver_master_addrs=127.24.123.62:44949
--webserver_port=45253
--webserver_interface=127.24.123.3
--builtin_ntp_servers=127.24.123.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:07.296458 8575 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44949
I20240503 08:14:07.296743 8575 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:07.297685 8575 heartbeater.cc:511] Master 127.24.123.62:44949 requested a full tablet report, sending...
I20240503 08:14:07.299482 7650 ts_manager.cc:194] Re-registered known tserver with Master: 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2:41059)
I20240503 08:14:07.300542 7650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.2:35621
W20240503 08:14:07.614915 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:07.615185 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:07.615296 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:07.622568 8579 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:07.622691 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:07.622839 8579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.3
I20240503 08:14:07.631436 8579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.3:36157
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.123.3
--webserver_port=45253
--tserver_master_addrs=127.24.123.62:44949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.3
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:07.632681 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:07.634279 8579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:07.646592 8587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:07.646961 8585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:07.647439 8584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:07.655309 8579 server_base.cc:1034] running on GCE node
I20240503 08:14:07.655985 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:07.657025 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:07.658258 8579 hybrid_clock.cc:648] HybridClock initialized: now 1714724047658203 us; error 47 us; skew 500 ppm
I20240503 08:14:07.658627 8579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:07.660883 8579 webserver.cc:460] Webserver started at http://127.24.123.3:45253/ using document root <none> and password file <none>
I20240503 08:14:07.661598 8579 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:07.661729 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:07.667008 8579 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:14:07.669597 8593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:07.670847 8579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:07.671044 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e46300ca67be4f5f9d1d84ce3d14136e"
format_stamp: "Formatted at 2024-05-03 08:13:59 on dist-test-slave-gkrw"
I20240503 08:14:07.671839 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:07.684643 8579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:07.685405 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:07.685670 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:07.686558 8579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:07.688016 8579 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:07.688093 8579 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:07.688205 8579 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:07.688252 8579 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:07.718215 8579 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.3:36157
I20240503 08:14:07.718254 8705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.3:36157 every 8 connection(s)
I20240503 08:14:07.719640 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:14:07.727169 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 8579
I20240503 08:14:07.732120 8706 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44949
I20240503 08:14:07.732467 8706 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:07.733433 8706 heartbeater.cc:511] Master 127.24.123.62:44949 requested a full tablet report, sending...
I20240503 08:14:07.734771 7650 ts_manager.cc:194] Re-registered known tserver with Master: e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3:36157)
I20240503 08:14:07.735525 7650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.3:50899
I20240503 08:14:08.182694 7780 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:08.188834 8173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:08.199956 8510 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:08.205765 8641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:08.303471 8575 heartbeater.cc:503] Master 127.24.123.62:44949 was elected leader, sending a full tablet report...
I20240503 08:14:08.344594 7780 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:08.345297 8510 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:08.346900 8173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:08.363993 8641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:08.517796 7780 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:08.522100 8510 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:08.532537 8173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:08.547641 8641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:08.673247 8510 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:08.673465 7780 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:08.686746 8641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:08.688418 8173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:08.737910 8706 heartbeater.cc:503] Master 127.24.123.62:44949 was elected leader, sending a full tablet report...
I20240503 08:14:08.754407 7800 consensus_queue.cc:237] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } }
I20240503 08:14:08.757077 8191 raft_consensus.cc:1261] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:08.757642 8844 consensus_queue.cc:1035] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:08.760385 8440 raft_consensus.cc:2932] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } } }
I20240503 08:14:08.760977 8191 raft_consensus.cc:2932] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } } }
W20240503 08:14:08.763099 7736 consensus_peers.cc:485] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 -> Peer 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2:41059): Couldn't send request to peer 256fc1afb22940258b553ca2cf763fd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e10ce115a9d4402bf4db426db4fbeac. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:08.766162 7650 catalog_manager.cc:5484] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: config changed from index -1 to 4, NON_VOTER 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) added. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } } }
I20240503 08:14:08.777290 8191 consensus_queue.cc:237] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } }
I20240503 08:14:08.779557 7800 raft_consensus.cc:1261] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:08.780076 8344 consensus_queue.cc:1035] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:08.783178 8857 raft_consensus.cc:2932] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } } }
I20240503 08:14:08.786077 7800 raft_consensus.cc:2932] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } } }
W20240503 08:14:08.787452 8129 consensus_peers.cc:485] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 -> Peer e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3:36157): Couldn't send request to peer e46300ca67be4f5f9d1d84ce3d14136e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90506a0036834a3180d6eaa0d030c1ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:08.791069 7656 catalog_manager.cc:5484] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: config changed from index -1 to 4, NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) added. New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:08.802044 7800 consensus_queue.cc:237] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } }
I20240503 08:14:08.803946 8191 raft_consensus.cc:1261] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:08.804366 7736 consensus_peers.cc:485] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 -> Peer 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2:41059): Couldn't send request to peer 256fc1afb22940258b553ca2cf763fd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b3f0c69966b4703b025e8f915d59c82. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:08.804405 8844 consensus_queue.cc:1035] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:08.806638 8440 raft_consensus.cc:2932] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } } }
I20240503 08:14:08.807215 8191 raft_consensus.cc:2932] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } } }
I20240503 08:14:08.814190 7656 catalog_manager.cc:5484] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: config changed from index -1 to 4, NON_VOTER 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) added. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } } }
I20240503 08:14:08.824028 8191 consensus_queue.cc:237] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } }
I20240503 08:14:08.825774 7800 raft_consensus.cc:1261] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:08.825915 8129 consensus_peers.cc:485] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 -> Peer e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3:36157): Couldn't send request to peer e46300ca67be4f5f9d1d84ce3d14136e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fec10fdaedf44a46abc4a3f970a4305c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:08.826191 8344 consensus_queue.cc:1035] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:08.828372 8857 raft_consensus.cc:2932] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } } }
I20240503 08:14:08.828915 7800 raft_consensus.cc:2932] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } } }
I20240503 08:14:08.832861 7650 catalog_manager.cc:5484] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: config changed from index -1 to 4, NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) added. New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:08.843343 7800 consensus_queue.cc:237] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } }
I20240503 08:14:08.845106 8191 raft_consensus.cc:1261] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:14:08.845253 7736 consensus_peers.cc:485] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 -> Peer 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2:41059): Couldn't send request to peer 256fc1afb22940258b553ca2cf763fd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 987da8ea2afa4f098e101a2e8c3ba53d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:08.845580 8844 consensus_queue.cc:1035] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:08.847743 8440 raft_consensus.cc:2932] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } } }
I20240503 08:14:08.848284 8191 raft_consensus.cc:2932] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } } }
I20240503 08:14:08.851794 7650 catalog_manager.cc:5484] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 reported cstate change: config changed from index -1 to 4, NON_VOTER 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) added. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:08.865909 7800 consensus_queue.cc:237] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } }
I20240503 08:14:08.868216 8191 raft_consensus.cc:1261] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:08.868991 8844 consensus_queue.cc:1035] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:08.871335 8867 raft_consensus.cc:2932] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } } }
I20240503 08:14:08.872001 8191 raft_consensus.cc:2932] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } } }
W20240503 08:14:08.874711 7736 consensus_peers.cc:485] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 -> Peer e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3:36157): Couldn't send request to peer e46300ca67be4f5f9d1d84ce3d14136e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 746748dba86242da9a54e1346ccfa2c9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:08.876796 7656 catalog_manager.cc:5484] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 reported cstate change: config changed from index -1 to 4, NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) added. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:09.188665 8949 ts_tablet_manager.cc:902] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: Initiating tablet copy from peer 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:09.189116 8950 ts_tablet_manager.cc:902] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: Initiating tablet copy from peer 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:09.190935 8949 tablet_copy_client.cc:308] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:43277
I20240503 08:14:09.191792 8950 tablet_copy_client.cc:308] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:37315
I20240503 08:14:09.198781 7820 tablet_copy_service.cc:140] P 087ad327ce194194ac083345d973b083: Received BeginTabletCopySession request for tablet 6e10ce115a9d4402bf4db426db4fbeac from peer 256fc1afb22940258b553ca2cf763fd5 ({username='slave'} at 127.24.123.2:56067)
I20240503 08:14:09.199129 7820 tablet_copy_service.cc:161] P 087ad327ce194194ac083345d973b083: Beginning new tablet copy session on tablet 6e10ce115a9d4402bf4db426db4fbeac from peer 256fc1afb22940258b553ca2cf763fd5 at {username='slave'} at 127.24.123.2:56067: session id = 256fc1afb22940258b553ca2cf763fd5-6e10ce115a9d4402bf4db426db4fbeac
I20240503 08:14:09.200807 8213 tablet_copy_service.cc:140] P 34a09bb831e64f278fd57493bf6554d1: Received BeginTabletCopySession request for tablet 90506a0036834a3180d6eaa0d030c1ec from peer e46300ca67be4f5f9d1d84ce3d14136e ({username='slave'} at 127.24.123.3:52853)
I20240503 08:14:09.201052 8213 tablet_copy_service.cc:161] P 34a09bb831e64f278fd57493bf6554d1: Beginning new tablet copy session on tablet 90506a0036834a3180d6eaa0d030c1ec from peer e46300ca67be4f5f9d1d84ce3d14136e at {username='slave'} at 127.24.123.3:52853: session id = e46300ca67be4f5f9d1d84ce3d14136e-90506a0036834a3180d6eaa0d030c1ec
I20240503 08:14:09.202967 7820 tablet_copy_source_session.cc:205] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:09.204946 8213 tablet_copy_source_session.cc:205] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:09.207989 8950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e10ce115a9d4402bf4db426db4fbeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:09.208938 8949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90506a0036834a3180d6eaa0d030c1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:09.217555 8950 tablet_copy_client.cc:783] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:09.217773 8949 tablet_copy_client.cc:783] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:09.217903 8950 tablet_copy_client.cc:647] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:09.218091 8949 tablet_copy_client.cc:647] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:09.220417 8950 tablet_copy_client.cc:523] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:09.220530 8949 tablet_copy_client.cc:523] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:09.223415 8949 tablet_bootstrap.cc:492] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: Bootstrap starting.
I20240503 08:14:09.223416 8950 tablet_bootstrap.cc:492] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: Bootstrap starting.
I20240503 08:14:09.231561 8950 log.cc:824] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:09.231559 8949 log.cc:824] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:09.248029 8963 ts_tablet_manager.cc:902] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: Initiating tablet copy from peer 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:09.261804 8965 ts_tablet_manager.cc:902] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: Initiating tablet copy from peer 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:09.272545 8950 tablet_bootstrap.cc:492] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:09.273288 8950 tablet_bootstrap.cc:492] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: Bootstrap complete.
I20240503 08:14:09.273727 8949 tablet_bootstrap.cc:492] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:09.273916 8950 ts_tablet_manager.cc:1367] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: Time spent bootstrapping tablet: real 0.051s user 0.046s sys 0.004s
I20240503 08:14:09.274425 8949 tablet_bootstrap.cc:492] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: Bootstrap complete.
I20240503 08:14:09.274969 8949 ts_tablet_manager.cc:1367] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: Time spent bootstrapping tablet: real 0.052s user 0.047s sys 0.004s
I20240503 08:14:09.281173 8950 raft_consensus.cc:348] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } }
I20240503 08:14:09.281806 8950 raft_consensus.cc:729] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 256fc1afb22940258b553ca2cf763fd5, State: Initialized, Role: LEARNER
I20240503 08:14:09.282459 8963 tablet_copy_client.cc:308] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:37315
I20240503 08:14:09.282462 8950 consensus_queue.cc:260] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } }
I20240503 08:14:09.284852 8965 tablet_copy_client.cc:308] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:37315
I20240503 08:14:09.285668 7820 tablet_copy_service.cc:140] P 087ad327ce194194ac083345d973b083: Received BeginTabletCopySession request for tablet 987da8ea2afa4f098e101a2e8c3ba53d from peer 256fc1afb22940258b553ca2cf763fd5 ({username='slave'} at 127.24.123.2:56067)
I20240503 08:14:09.285895 7820 tablet_copy_service.cc:161] P 087ad327ce194194ac083345d973b083: Beginning new tablet copy session on tablet 987da8ea2afa4f098e101a2e8c3ba53d from peer 256fc1afb22940258b553ca2cf763fd5 at {username='slave'} at 127.24.123.2:56067: session id = 256fc1afb22940258b553ca2cf763fd5-987da8ea2afa4f098e101a2e8c3ba53d
I20240503 08:14:09.289031 7820 tablet_copy_source_session.cc:205] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:09.290426 7820 tablet_copy_service.cc:140] P 087ad327ce194194ac083345d973b083: Received BeginTabletCopySession request for tablet 746748dba86242da9a54e1346ccfa2c9 from peer e46300ca67be4f5f9d1d84ce3d14136e ({username='slave'} at 127.24.123.3:42687)
I20240503 08:14:09.290560 7820 tablet_copy_service.cc:161] P 087ad327ce194194ac083345d973b083: Beginning new tablet copy session on tablet 746748dba86242da9a54e1346ccfa2c9 from peer e46300ca67be4f5f9d1d84ce3d14136e at {username='slave'} at 127.24.123.3:42687: session id = e46300ca67be4f5f9d1d84ce3d14136e-746748dba86242da9a54e1346ccfa2c9
I20240503 08:14:09.290669 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 987da8ea2afa4f098e101a2e8c3ba53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:09.293352 7820 tablet_copy_source_session.cc:205] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:09.294469 8963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746748dba86242da9a54e1346ccfa2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:09.298545 8965 tablet_copy_client.cc:783] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:09.298918 8965 tablet_copy_client.cc:647] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:09.299335 8949 raft_consensus.cc:348] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } }
I20240503 08:14:09.299916 8949 raft_consensus.cc:729] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e46300ca67be4f5f9d1d84ce3d14136e, State: Initialized, Role: LEARNER
I20240503 08:14:09.300560 8949 consensus_queue.cc:260] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } }
I20240503 08:14:09.300829 8963 tablet_copy_client.cc:783] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:09.301095 8963 tablet_copy_client.cc:647] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:09.302703 8950 ts_tablet_manager.cc:1398] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5: Time spent starting tablet: real 0.029s user 0.011s sys 0.016s
I20240503 08:14:09.303332 8963 tablet_copy_client.cc:523] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:09.305321 7820 tablet_copy_service.cc:342] P 087ad327ce194194ac083345d973b083: Request end of tablet copy session 256fc1afb22940258b553ca2cf763fd5-6e10ce115a9d4402bf4db426db4fbeac received from {username='slave'} at 127.24.123.2:56067
I20240503 08:14:09.305485 7820 tablet_copy_service.cc:433] P 087ad327ce194194ac083345d973b083: ending tablet copy session 256fc1afb22940258b553ca2cf763fd5-6e10ce115a9d4402bf4db426db4fbeac on tablet 6e10ce115a9d4402bf4db426db4fbeac with peer 256fc1afb22940258b553ca2cf763fd5
I20240503 08:14:09.305567 8949 ts_tablet_manager.cc:1398] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e: Time spent starting tablet: real 0.030s user 0.010s sys 0.001s
I20240503 08:14:09.306667 8965 tablet_copy_client.cc:523] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:09.306667 8213 tablet_copy_service.cc:342] P 34a09bb831e64f278fd57493bf6554d1: Request end of tablet copy session e46300ca67be4f5f9d1d84ce3d14136e-90506a0036834a3180d6eaa0d030c1ec received from {username='slave'} at 127.24.123.3:52853
I20240503 08:14:09.306753 8963 tablet_bootstrap.cc:492] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: Bootstrap starting.
I20240503 08:14:09.306834 8213 tablet_copy_service.cc:433] P 34a09bb831e64f278fd57493bf6554d1: ending tablet copy session e46300ca67be4f5f9d1d84ce3d14136e-90506a0036834a3180d6eaa0d030c1ec on tablet 90506a0036834a3180d6eaa0d030c1ec with peer e46300ca67be4f5f9d1d84ce3d14136e
I20240503 08:14:09.310963 8965 tablet_bootstrap.cc:492] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: Bootstrap starting.
I20240503 08:14:09.338294 8963 tablet_bootstrap.cc:492] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:09.338675 8963 tablet_bootstrap.cc:492] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: Bootstrap complete.
I20240503 08:14:09.338969 8963 ts_tablet_manager.cc:1367] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.011s
I20240503 08:14:09.339735 8963 raft_consensus.cc:348] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } }
I20240503 08:14:09.339887 8965 tablet_bootstrap.cc:492] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:09.339926 8963 raft_consensus.cc:729] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e46300ca67be4f5f9d1d84ce3d14136e, State: Initialized, Role: LEARNER
I20240503 08:14:09.340091 8963 consensus_queue.cc:260] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } }
I20240503 08:14:09.340222 8965 tablet_bootstrap.cc:492] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: Bootstrap complete.
I20240503 08:14:09.340462 8965 ts_tablet_manager.cc:1367] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.005s
I20240503 08:14:09.340569 8963 ts_tablet_manager.cc:1398] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:14:09.341378 7820 tablet_copy_service.cc:342] P 087ad327ce194194ac083345d973b083: Request end of tablet copy session e46300ca67be4f5f9d1d84ce3d14136e-746748dba86242da9a54e1346ccfa2c9 received from {username='slave'} at 127.24.123.3:42687
I20240503 08:14:09.341325 8965 raft_consensus.cc:348] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } }
I20240503 08:14:09.341523 7820 tablet_copy_service.cc:433] P 087ad327ce194194ac083345d973b083: ending tablet copy session e46300ca67be4f5f9d1d84ce3d14136e-746748dba86242da9a54e1346ccfa2c9 on tablet 746748dba86242da9a54e1346ccfa2c9 with peer e46300ca67be4f5f9d1d84ce3d14136e
I20240503 08:14:09.341552 8965 raft_consensus.cc:729] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 256fc1afb22940258b553ca2cf763fd5, State: Initialized, Role: LEARNER
I20240503 08:14:09.341744 8965 consensus_queue.cc:260] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } }
I20240503 08:14:09.343032 8965 ts_tablet_manager.cc:1398] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:14:09.343670 7820 tablet_copy_service.cc:342] P 087ad327ce194194ac083345d973b083: Request end of tablet copy session 256fc1afb22940258b553ca2cf763fd5-987da8ea2afa4f098e101a2e8c3ba53d received from {username='slave'} at 127.24.123.2:56067
I20240503 08:14:09.343808 7820 tablet_copy_service.cc:433] P 087ad327ce194194ac083345d973b083: ending tablet copy session 256fc1afb22940258b553ca2cf763fd5-987da8ea2afa4f098e101a2e8c3ba53d on tablet 987da8ea2afa4f098e101a2e8c3ba53d with peer 256fc1afb22940258b553ca2cf763fd5
I20240503 08:14:09.354846 8963 ts_tablet_manager.cc:902] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: Initiating tablet copy from peer 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277)
I20240503 08:14:09.355824 8963 tablet_copy_client.cc:308] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:43277
I20240503 08:14:09.356405 8213 tablet_copy_service.cc:140] P 34a09bb831e64f278fd57493bf6554d1: Received BeginTabletCopySession request for tablet fec10fdaedf44a46abc4a3f970a4305c from peer e46300ca67be4f5f9d1d84ce3d14136e ({username='slave'} at 127.24.123.3:52853)
I20240503 08:14:09.356554 8213 tablet_copy_service.cc:161] P 34a09bb831e64f278fd57493bf6554d1: Beginning new tablet copy session on tablet fec10fdaedf44a46abc4a3f970a4305c from peer e46300ca67be4f5f9d1d84ce3d14136e at {username='slave'} at 127.24.123.3:52853: session id = e46300ca67be4f5f9d1d84ce3d14136e-fec10fdaedf44a46abc4a3f970a4305c
I20240503 08:14:09.358386 8213 tablet_copy_source_session.cc:205] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:09.359470 8963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec10fdaedf44a46abc4a3f970a4305c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:09.363596 8963 tablet_copy_client.cc:783] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:09.363794 8963 tablet_copy_client.cc:647] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:09.365193 8963 tablet_copy_client.cc:523] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:09.367400 8963 tablet_bootstrap.cc:492] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: Bootstrap starting.
I20240503 08:14:09.400362 8963 tablet_bootstrap.cc:492] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:09.400738 8963 tablet_bootstrap.cc:492] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: Bootstrap complete.
I20240503 08:14:09.400961 8963 ts_tablet_manager.cc:1367] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.005s
I20240503 08:14:09.401651 8963 raft_consensus.cc:348] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } }
I20240503 08:14:09.401882 8963 raft_consensus.cc:729] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e46300ca67be4f5f9d1d84ce3d14136e, State: Initialized, Role: LEARNER
I20240503 08:14:09.402101 8963 consensus_queue.cc:260] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: true } }
I20240503 08:14:09.402679 8963 ts_tablet_manager.cc:1398] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:09.403479 8213 tablet_copy_service.cc:342] P 34a09bb831e64f278fd57493bf6554d1: Request end of tablet copy session e46300ca67be4f5f9d1d84ce3d14136e-fec10fdaedf44a46abc4a3f970a4305c received from {username='slave'} at 127.24.123.3:52853
I20240503 08:14:09.403605 8213 tablet_copy_service.cc:433] P 34a09bb831e64f278fd57493bf6554d1: ending tablet copy session e46300ca67be4f5f9d1d84ce3d14136e-fec10fdaedf44a46abc4a3f970a4305c on tablet fec10fdaedf44a46abc4a3f970a4305c with peer e46300ca67be4f5f9d1d84ce3d14136e
I20240503 08:14:09.426571 8965 ts_tablet_manager.cc:902] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: Initiating tablet copy from peer 087ad327ce194194ac083345d973b083 (127.24.123.1:37315)
I20240503 08:14:09.431543 8965 tablet_copy_client.cc:308] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:37315
I20240503 08:14:09.432197 7820 tablet_copy_service.cc:140] P 087ad327ce194194ac083345d973b083: Received BeginTabletCopySession request for tablet 5b3f0c69966b4703b025e8f915d59c82 from peer 256fc1afb22940258b553ca2cf763fd5 ({username='slave'} at 127.24.123.2:56067)
I20240503 08:14:09.432374 7820 tablet_copy_service.cc:161] P 087ad327ce194194ac083345d973b083: Beginning new tablet copy session on tablet 5b3f0c69966b4703b025e8f915d59c82 from peer 256fc1afb22940258b553ca2cf763fd5 at {username='slave'} at 127.24.123.2:56067: session id = 256fc1afb22940258b553ca2cf763fd5-5b3f0c69966b4703b025e8f915d59c82
I20240503 08:14:09.434492 7820 tablet_copy_source_session.cc:205] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:09.435523 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3f0c69966b4703b025e8f915d59c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:09.440460 8965 tablet_copy_client.cc:783] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:09.440666 8965 tablet_copy_client.cc:647] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:09.442112 8965 tablet_copy_client.cc:523] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:09.444396 8965 tablet_bootstrap.cc:492] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: Bootstrap starting.
I20240503 08:14:09.473418 8965 tablet_bootstrap.cc:492] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:09.473819 8965 tablet_bootstrap.cc:492] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: Bootstrap complete.
I20240503 08:14:09.474048 8965 ts_tablet_manager.cc:1367] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.001s
I20240503 08:14:09.474675 8965 raft_consensus.cc:348] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } }
I20240503 08:14:09.474851 8965 raft_consensus.cc:729] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 256fc1afb22940258b553ca2cf763fd5, State: Initialized, Role: LEARNER
I20240503 08:14:09.475070 8965 consensus_queue.cc:260] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: NON_VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: true } }
I20240503 08:14:09.475538 8965 ts_tablet_manager.cc:1398] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:14:09.476418 7820 tablet_copy_service.cc:342] P 087ad327ce194194ac083345d973b083: Request end of tablet copy session 256fc1afb22940258b553ca2cf763fd5-5b3f0c69966b4703b025e8f915d59c82 received from {username='slave'} at 127.24.123.2:56067
I20240503 08:14:09.476528 7820 tablet_copy_service.cc:433] P 087ad327ce194194ac083345d973b083: ending tablet copy session 256fc1afb22940258b553ca2cf763fd5-5b3f0c69966b4703b025e8f915d59c82 on tablet 5b3f0c69966b4703b025e8f915d59c82 with peer 256fc1afb22940258b553ca2cf763fd5
I20240503 08:14:09.635921 8530 raft_consensus.cc:1203] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:09.682859 8661 raft_consensus.cc:1203] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:09.776391 8661 raft_consensus.cc:1203] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:09.851384 8661 raft_consensus.cc:1203] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:09.859392 8530 raft_consensus.cc:1203] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:09.916095 8530 raft_consensus.cc:1203] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:14:10.093123 8867 raft_consensus.cc:1050] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083: attempting to promote NON_VOTER 256fc1afb22940258b553ca2cf763fd5 to VOTER
I20240503 08:14:10.094027 8867 consensus_queue.cc:237] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } }
I20240503 08:14:10.096247 8191 raft_consensus.cc:1261] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.096351 8530 raft_consensus.cc:1261] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.096767 9017 consensus_queue.cc:1035] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.097081 9018 consensus_queue.cc:1035] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.099896 9018 raft_consensus.cc:2932] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.100636 8191 raft_consensus.cc:2932] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.101385 8530 raft_consensus.cc:2932] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.105165 7650 catalog_manager.cc:5484] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 reported cstate change: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.111390 7799 consensus_queue.cc:237] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } }
I20240503 08:14:10.113517 8530 raft_consensus.cc:1261] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:10.114032 8867 consensus_queue.cc:1035] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:10.115898 9017 raft_consensus.cc:2932] T 6e10ce115a9d4402bf4db426db4fbeac P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.116456 8530 raft_consensus.cc:2932] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.119060 7645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e10ce115a9d4402bf4db426db4fbeac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:10.120179 7656 catalog_manager.cc:5484] T 6e10ce115a9d4402bf4db426db4fbeac P 256fc1afb22940258b553ca2cf763fd5 reported cstate change: config changed from index 5 to 6, VOTER 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.123366 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e10ce115a9d4402bf4db426db4fbeac with delete_type TABLET_DATA_TOMBSTONED (TS 34a09bb831e64f278fd57493bf6554d1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58912
I20240503 08:14:10.124183 9088 tablet_replica.cc:330] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1: stopping tablet replica
I20240503 08:14:10.124652 9088 raft_consensus.cc:2229] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:10.125051 9088 raft_consensus.cc:2258] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:10.126356 9088 ts_tablet_manager.cc:1875] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:10.130610 9088 ts_tablet_manager.cc:1888] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:10.130734 9088 log.cc:1196] T 6e10ce115a9d4402bf4db426db4fbeac P 34a09bb831e64f278fd57493bf6554d1: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/6e10ce115a9d4402bf4db426db4fbeac
I20240503 08:14:10.131496 7645 catalog_manager.cc:4830] TS 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277): tablet 6e10ce115a9d4402bf4db426db4fbeac (table rebalance_test_table_2 [id=26b2dfa77389481e8c16612262154ee6]) successfully deleted
I20240503 08:14:10.151096 9045 raft_consensus.cc:1050] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1: attempting to promote NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e to VOTER
I20240503 08:14:10.151816 9045 consensus_queue.cc:237] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } }
I20240503 08:14:10.153950 8661 raft_consensus.cc:1261] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.154194 7799 raft_consensus.cc:1261] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.154559 9045 consensus_queue.cc:1035] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.154950 8344 consensus_queue.cc:1035] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.157833 9024 raft_consensus.cc:2932] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.158547 7799 raft_consensus.cc:2932] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.159920 8661 raft_consensus.cc:2932] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.162964 7656 catalog_manager.cc:5484] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.168377 8191 consensus_queue.cc:237] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } }
I20240503 08:14:10.170183 8661 raft_consensus.cc:1261] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e [term 1 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:10.170658 8344 consensus_queue.cc:1035] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:10.172812 9045 raft_consensus.cc:2932] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 087ad327ce194194ac083345d973b083 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.173336 8661 raft_consensus.cc:2932] T 90506a0036834a3180d6eaa0d030c1ec P e46300ca67be4f5f9d1d84ce3d14136e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 087ad327ce194194ac083345d973b083 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.175313 7645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 90506a0036834a3180d6eaa0d030c1ec with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:10.177070 7653 catalog_manager.cc:5484] T 90506a0036834a3180d6eaa0d030c1ec P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: config changed from index 5 to 6, VOTER 087ad327ce194194ac083345d973b083 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.180379 7780 tablet_service.cc:1514] Processing DeleteTablet for tablet 90506a0036834a3180d6eaa0d030c1ec with delete_type TABLET_DATA_TOMBSTONED (TS 087ad327ce194194ac083345d973b083 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33838
I20240503 08:14:10.181218 9104 tablet_replica.cc:330] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083: stopping tablet replica
I20240503 08:14:10.181728 9104 raft_consensus.cc:2229] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:10.182196 9104 raft_consensus.cc:2258] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:10.183753 9104 ts_tablet_manager.cc:1875] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:10.190343 9104 ts_tablet_manager.cc:1888] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:10.190454 9104 log.cc:1196] T 90506a0036834a3180d6eaa0d030c1ec P 087ad327ce194194ac083345d973b083: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/90506a0036834a3180d6eaa0d030c1ec
I20240503 08:14:10.191190 7645 catalog_manager.cc:4830] TS 087ad327ce194194ac083345d973b083 (127.24.123.1:37315): tablet 90506a0036834a3180d6eaa0d030c1ec (table rebalance_test_table_1 [id=beeccc26a3874634b97dd0b996caa598]) successfully deleted
I20240503 08:14:10.252067 8344 raft_consensus.cc:1050] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1: attempting to promote NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e to VOTER
I20240503 08:14:10.252647 8344 consensus_queue.cc:237] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } }
I20240503 08:14:10.254371 8661 raft_consensus.cc:1261] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.254388 7799 raft_consensus.cc:1261] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.254864 9024 consensus_queue.cc:1035] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.255177 8344 consensus_queue.cc:1035] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.257483 9045 raft_consensus.cc:2932] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.258028 7799 raft_consensus.cc:2932] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.260402 8661 raft_consensus.cc:2932] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.261711 7656 catalog_manager.cc:5484] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.265830 8191 consensus_queue.cc:237] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } }
I20240503 08:14:10.267186 8661 raft_consensus.cc:1261] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e [term 1 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:10.267683 8344 consensus_queue.cc:1035] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:10.269461 9045 raft_consensus.cc:2932] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 087ad327ce194194ac083345d973b083 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.270071 8661 raft_consensus.cc:2932] T fec10fdaedf44a46abc4a3f970a4305c P e46300ca67be4f5f9d1d84ce3d14136e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 087ad327ce194194ac083345d973b083 (127.24.123.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.271927 7645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fec10fdaedf44a46abc4a3f970a4305c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:10.273538 7650 catalog_manager.cc:5484] T fec10fdaedf44a46abc4a3f970a4305c P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: config changed from index 5 to 6, VOTER 087ad327ce194194ac083345d973b083 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.276271 7780 tablet_service.cc:1514] Processing DeleteTablet for tablet fec10fdaedf44a46abc4a3f970a4305c with delete_type TABLET_DATA_TOMBSTONED (TS 087ad327ce194194ac083345d973b083 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33838
I20240503 08:14:10.276619 9104 tablet_replica.cc:330] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083: stopping tablet replica
I20240503 08:14:10.276820 9104 raft_consensus.cc:2229] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:10.277158 9104 raft_consensus.cc:2258] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:10.277956 9104 ts_tablet_manager.cc:1875] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:10.282025 9104 ts_tablet_manager.cc:1888] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:10.282140 9104 log.cc:1196] T fec10fdaedf44a46abc4a3f970a4305c P 087ad327ce194194ac083345d973b083: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/fec10fdaedf44a46abc4a3f970a4305c
I20240503 08:14:10.282745 7645 catalog_manager.cc:4830] TS 087ad327ce194194ac083345d973b083 (127.24.123.1:37315): tablet fec10fdaedf44a46abc4a3f970a4305c (table rebalance_test_table_0 [id=ef773cd915eb4ae7bd04beb5c55ca9af]) successfully deleted
I20240503 08:14:10.319358 8867 raft_consensus.cc:1050] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083: attempting to promote NON_VOTER 256fc1afb22940258b553ca2cf763fd5 to VOTER
I20240503 08:14:10.319608 9018 raft_consensus.cc:1050] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083: attempting to promote NON_VOTER 256fc1afb22940258b553ca2cf763fd5 to VOTER
I20240503 08:14:10.320099 8867 consensus_queue.cc:237] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } }
I20240503 08:14:10.320183 9018 consensus_queue.cc:237] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } }
I20240503 08:14:10.322275 8529 raft_consensus.cc:1261] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.322822 8191 raft_consensus.cc:1261] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.323269 8193 raft_consensus.cc:1261] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.323829 9017 consensus_queue.cc:1035] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:14:10.324167 8867 consensus_queue.cc:1035] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.324316 8529 raft_consensus.cc:1261] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5 [term 1 LEARNER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.324512 9146 consensus_queue.cc:1035] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.325054 9146 consensus_queue.cc:1035] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.327678 9018 raft_consensus.cc:2932] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.328514 8193 raft_consensus.cc:2932] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.328610 8530 raft_consensus.cc:2932] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.332558 9017 raft_consensus.cc:2932] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.332984 7656 catalog_manager.cc:5484] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 reported cstate change: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.333349 8530 raft_consensus.cc:2932] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.337697 7799 consensus_queue.cc:237] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } }
I20240503 08:14:10.339174 8530 raft_consensus.cc:1261] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:10.339152 7650 catalog_manager.cc:5484] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 reported cstate change: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.339987 9150 consensus_queue.cc:1035] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:10.342332 9017 raft_consensus.cc:2932] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.343849 7799 consensus_queue.cc:237] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } }
I20240503 08:14:10.344342 8191 raft_consensus.cc:2932] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 256fc1afb22940258b553ca2cf763fd5 (127.24.123.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } attrs { replace: true } } peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.345567 8529 raft_consensus.cc:1261] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:10.345877 7645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b3f0c69966b4703b025e8f915d59c82 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:10.346026 8530 raft_consensus.cc:2932] T 5b3f0c69966b4703b025e8f915d59c82 P 256fc1afb22940258b553ca2cf763fd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.347000 9018 consensus_queue.cc:1035] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:10.348253 7650 catalog_manager.cc:5484] T 5b3f0c69966b4703b025e8f915d59c82 P 087ad327ce194194ac083345d973b083 reported cstate change: config changed from index 5 to 6, VOTER 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.348968 9150 raft_consensus.cc:2932] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.352286 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f0c69966b4703b025e8f915d59c82 with delete_type TABLET_DATA_TOMBSTONED (TS 34a09bb831e64f278fd57493bf6554d1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58912
I20240503 08:14:10.352602 9088 tablet_replica.cc:330] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: stopping tablet replica
I20240503 08:14:10.352800 9088 raft_consensus.cc:2229] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:10.352962 7645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 987da8ea2afa4f098e101a2e8c3ba53d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:10.353016 9088 raft_consensus.cc:2258] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:10.353694 8530 raft_consensus.cc:2932] T 987da8ea2afa4f098e101a2e8c3ba53d P 256fc1afb22940258b553ca2cf763fd5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } } }
I20240503 08:14:10.353983 9088 ts_tablet_manager.cc:1875] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:10.354465 7650 catalog_manager.cc:5484] T 987da8ea2afa4f098e101a2e8c3ba53d P 087ad327ce194194ac083345d973b083 reported cstate change: config changed from index 5 to 6, VOTER 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "256fc1afb22940258b553ca2cf763fd5" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 41059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.358006 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 987da8ea2afa4f098e101a2e8c3ba53d with delete_type TABLET_DATA_TOMBSTONED (TS 34a09bb831e64f278fd57493bf6554d1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58912
I20240503 08:14:10.359581 9088 ts_tablet_manager.cc:1888] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:10.359699 9088 log.cc:1196] T 5b3f0c69966b4703b025e8f915d59c82 P 34a09bb831e64f278fd57493bf6554d1: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/5b3f0c69966b4703b025e8f915d59c82
I20240503 08:14:10.360216 9088 tablet_replica.cc:330] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1: stopping tablet replica
I20240503 08:14:10.360258 7645 catalog_manager.cc:4830] TS 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277): tablet 5b3f0c69966b4703b025e8f915d59c82 (table rebalance_test_table_0 [id=ef773cd915eb4ae7bd04beb5c55ca9af]) successfully deleted
I20240503 08:14:10.360394 9088 raft_consensus.cc:2229] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:10.360570 9088 raft_consensus.cc:2258] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:10.361204 9088 ts_tablet_manager.cc:1875] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:10.364943 9088 ts_tablet_manager.cc:1888] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:14:10.365044 9088 log.cc:1196] T 987da8ea2afa4f098e101a2e8c3ba53d P 34a09bb831e64f278fd57493bf6554d1: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/987da8ea2afa4f098e101a2e8c3ba53d
I20240503 08:14:10.365566 7645 catalog_manager.cc:4830] TS 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4:43277): tablet 987da8ea2afa4f098e101a2e8c3ba53d (table rebalance_test_table_1 [id=beeccc26a3874634b97dd0b996caa598]) successfully deleted
I20240503 08:14:10.461755 9017 raft_consensus.cc:1050] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: attempting to promote NON_VOTER e46300ca67be4f5f9d1d84ce3d14136e to VOTER
I20240503 08:14:10.462548 9017 consensus_queue.cc:237] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } }
I20240503 08:14:10.464525 8661 raft_consensus.cc:1261] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 1 LEARNER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.464649 8191 raft_consensus.cc:1261] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Refusing update from remote peer 087ad327ce194194ac083345d973b083: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:10.465194 9146 consensus_queue.cc:1035] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.465528 9018 consensus_queue.cc:1035] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:10.468268 9146 raft_consensus.cc:2932] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.468989 8661 raft_consensus.cc:2932] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.472024 8193 raft_consensus.cc:2932] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:10.474041 7650 catalog_manager.cc:5484] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 reported cstate change: config changed from index 4 to 5, e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "087ad327ce194194ac083345d973b083" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:10.511838 7799 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "746748dba86242da9a54e1346ccfa2c9"
dest_uuid: "087ad327ce194194ac083345d973b083"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43572
I20240503 08:14:10.512100 7799 raft_consensus.cc:595] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:14:10.730487 7799 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "746748dba86242da9a54e1346ccfa2c9"
dest_uuid: "087ad327ce194194ac083345d973b083"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43594
I20240503 08:14:10.730733 7799 raft_consensus.cc:595] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:14:10.926333 9146 raft_consensus.cc:979] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: : Instructing follower 34a09bb831e64f278fd57493bf6554d1 to start an election
I20240503 08:14:10.926501 9146 raft_consensus.cc:1067] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Signalling peer 34a09bb831e64f278fd57493bf6554d1 to start an election
I20240503 08:14:10.927423 8193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "746748dba86242da9a54e1346ccfa2c9"
dest_uuid: "34a09bb831e64f278fd57493bf6554d1"
from {username='slave'} at 127.24.123.1:40273
I20240503 08:14:10.927659 8193 raft_consensus.cc:482] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:14:10.927820 8193 raft_consensus.cc:3037] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:10.929728 8193 raft_consensus.cc:504] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } }
I20240503 08:14:10.930362 8193 leader_election.cc:288] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 2 election: Requested vote from peers 087ad327ce194194ac083345d973b083 (127.24.123.1:37315), e46300ca67be4f5f9d1d84ce3d14136e (127.24.123.3:36157)
I20240503 08:14:10.930840 7799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746748dba86242da9a54e1346ccfa2c9" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "087ad327ce194194ac083345d973b083"
I20240503 08:14:10.931093 7799 raft_consensus.cc:3032] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:14:10.931028 8661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746748dba86242da9a54e1346ccfa2c9" candidate_uuid: "34a09bb831e64f278fd57493bf6554d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e46300ca67be4f5f9d1d84ce3d14136e"
I20240503 08:14:10.931178 7799 raft_consensus.cc:729] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 087ad327ce194194ac083345d973b083, State: Running, Role: LEADER
I20240503 08:14:10.931351 8661 raft_consensus.cc:3037] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:10.931492 7799 consensus_queue.cc:260] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } }
I20240503 08:14:10.931911 7799 raft_consensus.cc:3037] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:10.934258 8661 raft_consensus.cc:2454] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 2.
I20240503 08:14:10.934412 7799 raft_consensus.cc:2454] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a09bb831e64f278fd57493bf6554d1 in term 2.
I20240503 08:14:10.934813 8129 leader_election.cc:302] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a09bb831e64f278fd57493bf6554d1, e46300ca67be4f5f9d1d84ce3d14136e; no voters:
I20240503 08:14:10.935142 9045 raft_consensus.cc:2781] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:10.935503 9045 raft_consensus.cc:686] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 2 LEADER]: Becoming Leader. State: Replica: 34a09bb831e64f278fd57493bf6554d1, State: Running, Role: LEADER
I20240503 08:14:10.935829 9045 consensus_queue.cc:237] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } }
I20240503 08:14:10.938324 7656 catalog_manager.cc:5484] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: term changed from 1 to 2, leader changed from 087ad327ce194194ac083345d973b083 (127.24.123.1) to 34a09bb831e64f278fd57493bf6554d1 (127.24.123.4). New cstate: current_term: 2 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:11.412158 7799 raft_consensus.cc:1261] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 2 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:11.412817 9045 consensus_queue.cc:1035] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087ad327ce194194ac083345d973b083" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 37315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:11.416296 8661 raft_consensus.cc:1261] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 2 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:14:11.416927 9045 consensus_queue.cc:1035] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:11.967046 8193 consensus_queue.cc:237] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } }
I20240503 08:14:11.968528 8661 raft_consensus.cc:1261] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 2 FOLLOWER]: Refusing update from remote peer 34a09bb831e64f278fd57493bf6554d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:14:11.969008 9231 consensus_queue.cc:1035] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:11.970908 9230 raft_consensus.cc:2932] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 087ad327ce194194ac083345d973b083 (127.24.123.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:11.971491 8661 raft_consensus.cc:2932] T 746748dba86242da9a54e1346ccfa2c9 P e46300ca67be4f5f9d1d84ce3d14136e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 087ad327ce194194ac083345d973b083 (127.24.123.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } } }
I20240503 08:14:11.973594 7645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 746748dba86242da9a54e1346ccfa2c9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:11.975085 7653 catalog_manager.cc:5484] T 746748dba86242da9a54e1346ccfa2c9 P 34a09bb831e64f278fd57493bf6554d1 reported cstate change: config changed from index 5 to 7, VOTER 087ad327ce194194ac083345d973b083 (127.24.123.1) evicted. New cstate: current_term: 2 leader_uuid: "34a09bb831e64f278fd57493bf6554d1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34a09bb831e64f278fd57493bf6554d1" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46300ca67be4f5f9d1d84ce3d14136e" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 36157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:11.979511 7780 tablet_service.cc:1514] Processing DeleteTablet for tablet 746748dba86242da9a54e1346ccfa2c9 with delete_type TABLET_DATA_TOMBSTONED (TS 087ad327ce194194ac083345d973b083 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33838
I20240503 08:14:11.980085 9237 tablet_replica.cc:330] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: stopping tablet replica
I20240503 08:14:11.980260 9237 raft_consensus.cc:2229] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:11.980432 9237 raft_consensus.cc:2258] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:11.981215 9237 ts_tablet_manager.cc:1875] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:11.985757 9237 ts_tablet_manager.cc:1888] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:14:11.985862 9237 log.cc:1196] T 746748dba86242da9a54e1346ccfa2c9 P 087ad327ce194194ac083345d973b083: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/746748dba86242da9a54e1346ccfa2c9
I20240503 08:14:11.986678 7645 catalog_manager.cc:4830] TS 087ad327ce194194ac083345d973b083 (127.24.123.1:37315): tablet 746748dba86242da9a54e1346ccfa2c9 (table rebalance_test_table_2 [id=26b2dfa77389481e8c16612262154ee6]) successfully deleted
I20240503 08:14:12.093289 8510 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:12.100229 7780 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:12.101696 8173 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:12.122853 8641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:12.281364 8510 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:12.282734 7780 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:12.288865 8173 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:12.291355 8641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:12.425395 7780 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:12.427131 8173 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:12.436610 8510 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:12.438385 8641 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:12.609359 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 7718
I20240503 08:14:12.624881 7840 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:12.757962 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 7718
W20240503 08:14:12.776562 8465 connection.cc:535] server connection from 127.24.123.1:33987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:12.777222 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 8448
I20240503 08:14:12.783586 8569 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:14:12.798401 8128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.123.1:37315: connect: Connection refused (error 111)
W20240503 08:14:12.799902 8128 consensus_peers.cc:485] T 4feffc41ba284c7787642d3497c19a4e P 34a09bb831e64f278fd57493bf6554d1 -> Peer 087ad327ce194194ac083345d973b083 (127.24.123.1:37315): Couldn't send request to peer 087ad327ce194194ac083345d973b083. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:37315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:12.892268 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 8448
I20240503 08:14:12.909418 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 8579
I20240503 08:14:12.915478 8700 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:13.019125 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 8579
I20240503 08:14:13.039695 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 8110
I20240503 08:14:13.045569 8232 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:14:13.169714 8128 consensus_peers.cc:485] T d761e683bf8b4e3f853f42ae38a2a939 P 34a09bb831e64f278fd57493bf6554d1 -> Peer 087ad327ce194194ac083345d973b083 (127.24.123.1:37315): Couldn't send request to peer 087ad327ce194194ac083345d973b083. Status: Network error: Client connection negotiation failed: client connection to 127.24.123.1:37315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:13.171412 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 8110
I20240503 08:14:13.190124 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 7628
I20240503 08:14:13.191256 7689 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:13.319417 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 7628
2024-05-03T08:14:13Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (15814 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx6_rf7_t7_l_3_4_3
I20240503 08:14:13.364750 25068 test_util.cc:268] Using random seed: 1310109670
I20240503 08:14:13.366556 25068 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:14:13.366634 25068 ts_itest-base.cc:113] --------------
I20240503 08:14:13.366698 25068 ts_itest-base.cc:114] 10 tablet servers
I20240503 08:14:13.366765 25068 ts_itest-base.cc:115] 7 replicas per TS
I20240503 08:14:13.366813 25068 ts_itest-base.cc:116] --------------
2024-05-03T08:14:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:13Z Disabled control of system clock
I20240503 08:14:13.408893 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:44669
--webserver_interface=127.24.123.62
--webserver_port=0
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.123.62:44669
--location_mapping_cmd=/tmp/dist-test-taskEUWrla/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 08:14:13.732474 9363 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:14:13.732815 9363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:13.732913 9363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:13.732998 9363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:13.740725 9363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:13.740864 9363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:13.740968 9363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:13.741052 9363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:13.741140 9363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:13.751701 9363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskEUWrla/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--ipki_ca_key_size=768
--master_addresses=127.24.123.62:44669
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.123.62:44669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.123.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:13.753284 9363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:13.755131 9363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:13.766029 9368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:13.766445 9371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:13.767072 9369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:13.767560 9363 server_base.cc:1034] running on GCE node
I20240503 08:14:13.768337 9363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:13.769790 9363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:13.770982 9363 hybrid_clock.cc:648] HybridClock initialized: now 1714724053770921 us; error 48 us; skew 500 ppm
I20240503 08:14:13.771332 9363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:13.773579 9363 webserver.cc:460] Webserver started at http://127.24.123.62:42695/ using document root <none> and password file <none>
I20240503 08:14:13.774314 9363 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:13.774447 9363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:13.774734 9363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:13.777333 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a03d7e7827fc4d13abe43f01375f614c"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-gkrw"
I20240503 08:14:13.778103 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a03d7e7827fc4d13abe43f01375f614c"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-gkrw"
I20240503 08:14:13.783242 9363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240503 08:14:13.786473 9377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.787873 9363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:13.788105 9363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a03d7e7827fc4d13abe43f01375f614c"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-gkrw"
I20240503 08:14:13.788288 9363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:13.818152 9363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:13.819036 9363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:13.819316 9363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:13.836165 9363 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.62:44669
I20240503 08:14:13.836200 9428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.62:44669 every 8 connection(s)
I20240503 08:14:13.837467 9363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:14:13.841008 9429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:13.844991 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 9363
I20240503 08:14:13.845212 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:14:13.850903 9429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c: Bootstrap starting.
I20240503 08:14:13.854142 9429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:13.855410 9429 log.cc:824] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:13.858063 9429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c: No bootstrap required, opened a new log
I20240503 08:14:13.864120 9429 raft_consensus.cc:348] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03d7e7827fc4d13abe43f01375f614c" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44669 } }
I20240503 08:14:13.864418 9429 raft_consensus.cc:374] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:13.864488 9429 raft_consensus.cc:729] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03d7e7827fc4d13abe43f01375f614c, State: Initialized, Role: FOLLOWER
I20240503 08:14:13.865028 9429 consensus_queue.cc:260] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03d7e7827fc4d13abe43f01375f614c" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44669 } }
I20240503 08:14:13.865249 9429 raft_consensus.cc:388] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:13.865373 9429 raft_consensus.cc:482] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:13.865522 9429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:13.867656 9429 raft_consensus.cc:504] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03d7e7827fc4d13abe43f01375f614c" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44669 } }
I20240503 08:14:13.868085 9429 leader_election.cc:302] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a03d7e7827fc4d13abe43f01375f614c; no voters:
I20240503 08:14:13.868605 9429 leader_election.cc:288] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:14:13.868731 9434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:13.869313 9434 raft_consensus.cc:686] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [term 1 LEADER]: Becoming Leader. State: Replica: a03d7e7827fc4d13abe43f01375f614c, State: Running, Role: LEADER
I20240503 08:14:13.869794 9434 consensus_queue.cc:237] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03d7e7827fc4d13abe43f01375f614c" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44669 } }
I20240503 08:14:13.870456 9429 sys_catalog.cc:564] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:13.876354 9436 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a03d7e7827fc4d13abe43f01375f614c. Latest consensus state: current_term: 1 leader_uuid: "a03d7e7827fc4d13abe43f01375f614c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03d7e7827fc4d13abe43f01375f614c" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44669 } } }
I20240503 08:14:13.876693 9436 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:13.877583 9435 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a03d7e7827fc4d13abe43f01375f614c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03d7e7827fc4d13abe43f01375f614c" member_type: VOTER last_known_addr { host: "127.24.123.62" port: 44669 } } }
I20240503 08:14:13.877887 9435 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c [sys.catalog]: This master's current role is: LEADER
W20240503 08:14:13.885789 9448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:14:13.885901 9448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:14:13.886030 9449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:13.888098 9449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:13.896624 9449 catalog_manager.cc:1347] Generated new cluster ID: ae8aff767e194d27b383ea8486299b6a
I20240503 08:14:13.896752 9449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:13.919606 9449 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:13.920920 9449 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:13.930425 9449 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a03d7e7827fc4d13abe43f01375f614c: Generated new TSK 0
I20240503 08:14:13.931335 9449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:14:13.945065 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.1:0
--local_ip_for_outbound_sockets=127.24.123.1
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:14:14.284096 9453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:14.284364 9453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:14.284487 9453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:14.291783 9453 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:14.291900 9453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:14.292058 9453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.1
I20240503 08:14:14.301036 9453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.123.1
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.1
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:14.302349 9453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:14.304059 9453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:14.315559 9458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:14.315969 9461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:14.316274 9459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:14.317059 9453 server_base.cc:1034] running on GCE node
I20240503 08:14:14.317569 9453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:14.318408 9453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:14.319587 9453 hybrid_clock.cc:648] HybridClock initialized: now 1714724054319543 us; error 38 us; skew 500 ppm
I20240503 08:14:14.319859 9453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:14.321877 9453 webserver.cc:460] Webserver started at http://127.24.123.1:39235/ using document root <none> and password file <none>
I20240503 08:14:14.322626 9453 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:14.322727 9453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:14.323033 9453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:14.325528 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b3d3436263d04c57b2d47cc9d766efa5"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-gkrw"
I20240503 08:14:14.326208 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b3d3436263d04c57b2d47cc9d766efa5"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-gkrw"
I20240503 08:14:14.330792 9453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20240503 08:14:14.333922 9467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.335227 9453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:14.335361 9453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b3d3436263d04c57b2d47cc9d766efa5"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-gkrw"
I20240503 08:14:14.335520 9453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:14.351586 9453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:14.352326 9453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:14.352563 9453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:14.353415 9453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:14.354811 9453 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:14.354930 9453 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.355038 9453 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:14.355108 9453 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.383749 9453 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.1:39353
I20240503 08:14:14.383813 9579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.1:39353 every 8 connection(s)
I20240503 08:14:14.384974 9453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:14:14.385876 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 9453
I20240503 08:14:14.386088 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:14:14.389793 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.2:0
--local_ip_for_outbound_sockets=127.24.123.2
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:14.397810 9580 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:14.398145 9580 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:14.399196 9580 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:14.450013 9394 ts_manager.cc:194] Registered new tserver with Master: b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:14.452760 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.1:37601
W20240503 08:14:14.717584 9584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:14.717864 9584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:14.717955 9584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:14.725183 9584 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:14.725297 9584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:14.725421 9584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.2
I20240503 08:14:14.734324 9584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.123.2
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.2
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:14.735615 9584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:14.737247 9584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:14.748950 9591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:14.749325 9590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:14.749687 9593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:14.750550 9584 server_base.cc:1034] running on GCE node
I20240503 08:14:14.751214 9584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:14.752166 9584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:14.753367 9584 hybrid_clock.cc:648] HybridClock initialized: now 1714724054753316 us; error 50 us; skew 500 ppm
I20240503 08:14:14.753669 9584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:14.755982 9584 webserver.cc:460] Webserver started at http://127.24.123.2:36763/ using document root <none> and password file <none>
I20240503 08:14:14.756683 9584 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:14.756799 9584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:14.757097 9584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:14.759637 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-gkrw"
I20240503 08:14:14.760356 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-gkrw"
I20240503 08:14:14.765295 9584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:14:14.768663 9599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.769948 9584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 08:14:14.770098 9584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-gkrw"
I20240503 08:14:14.770273 9584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:14.786715 9584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:14.787539 9584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:14.787801 9584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:14.788690 9584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:14.790144 9584 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:14.790238 9584 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.790349 9584 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:14.790414 9584 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.822134 9584 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.2:36123
I20240503 08:14:14.822202 9711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.2:36123 every 8 connection(s)
I20240503 08:14:14.823493 9584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:14:14.829219 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 9584
I20240503 08:14:14.829419 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:14:14.832662 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.3:0
--local_ip_for_outbound_sockets=127.24.123.3
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:14.836180 9712 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:14.836486 9712 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:14.837427 9712 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:14.881520 9394 ts_manager.cc:194] Registered new tserver with Master: d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123)
I20240503 08:14:14.882730 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.2:58051
W20240503 08:14:15.163666 9715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:15.163939 9715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:15.164042 9715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:15.171551 9715 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:15.171674 9715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:15.171789 9715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.3
I20240503 08:14:15.180817 9715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.123.3
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.3
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:15.182056 9715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:15.183727 9715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:15.195744 9721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.196206 9724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.196710 9722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:15.197248 9715 server_base.cc:1034] running on GCE node
I20240503 08:14:15.197826 9715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:15.198700 9715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:15.199884 9715 hybrid_clock.cc:648] HybridClock initialized: now 1714724055199838 us; error 44 us; skew 500 ppm
I20240503 08:14:15.200174 9715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:15.206349 9715 webserver.cc:460] Webserver started at http://127.24.123.3:39423/ using document root <none> and password file <none>
I20240503 08:14:15.207134 9715 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.207245 9715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.207544 9715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:15.210443 9715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8de08e679a904ee48ba659861c730725"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-gkrw"
I20240503 08:14:15.211304 9715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8de08e679a904ee48ba659861c730725"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-gkrw"
I20240503 08:14:15.216629 9715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 08:14:15.220196 9730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.221503 9715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:14:15.221702 9715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8de08e679a904ee48ba659861c730725"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-gkrw"
I20240503 08:14:15.221971 9715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:15.241858 9715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:15.242769 9715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:15.243147 9715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:15.244073 9715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:15.245507 9715 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:15.245604 9715 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.245707 9715 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:15.245774 9715 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.273995 9715 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.3:41805
I20240503 08:14:15.274044 9842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.3:41805 every 8 connection(s)
I20240503 08:14:15.275362 9715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:14:15.276702 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 9715
I20240503 08:14:15.276854 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:14:15.279920 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.4:0
--local_ip_for_outbound_sockets=127.24.123.4
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:15.287742 9843 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:15.288033 9843 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:15.288815 9843 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:15.332664 9394 ts_manager.cc:194] Registered new tserver with Master: 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:15.333583 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.3:60491
I20240503 08:14:15.456466 9580 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
W20240503 08:14:15.610105 9846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:15.610385 9846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:15.610478 9846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:15.617736 9846 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:15.617851 9846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:15.617962 9846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.4
I20240503 08:14:15.626655 9846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.123.4
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.4
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:15.627956 9846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:15.629511 9846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:15.640744 9852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.641287 9853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.641697 9855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:15.642319 9846 server_base.cc:1034] running on GCE node
I20240503 08:14:15.642956 9846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:15.643846 9846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:15.645035 9846 hybrid_clock.cc:648] HybridClock initialized: now 1714724055644988 us; error 38 us; skew 500 ppm
I20240503 08:14:15.645325 9846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:15.647480 9846 webserver.cc:460] Webserver started at http://127.24.123.4:45231/ using document root <none> and password file <none>
I20240503 08:14:15.648173 9846 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.648277 9846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.648564 9846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:15.651089 9846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "06f1b17731834350a289de93abab8542"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-gkrw"
I20240503 08:14:15.651815 9846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "06f1b17731834350a289de93abab8542"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-gkrw"
I20240503 08:14:15.656600 9846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:14:15.659786 9861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.661120 9846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:14:15.661280 9846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "06f1b17731834350a289de93abab8542"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-gkrw"
I20240503 08:14:15.661468 9846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:15.694707 9846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:15.695591 9846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:15.695860 9846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:15.696810 9846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:15.698282 9846 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:15.698375 9846 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.698477 9846 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:15.698544 9846 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.727059 9846 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.4:38565
I20240503 08:14:15.727134 9973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.4:38565 every 8 connection(s)
I20240503 08:14:15.728327 9846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:14:15.734290 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 9846
I20240503 08:14:15.734505 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:14:15.737603 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.5:0
--local_ip_for_outbound_sockets=127.24.123.5
--webserver_interface=127.24.123.5
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:15.740989 9974 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:15.741309 9974 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:15.742112 9974 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:15.786119 9394 ts_manager.cc:194] Registered new tserver with Master: 06f1b17731834350a289de93abab8542 (127.24.123.4:38565)
I20240503 08:14:15.787422 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.4:56427
I20240503 08:14:15.885427 9712 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
W20240503 08:14:16.064173 9977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:16.064450 9977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:16.064563 9977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:16.072048 9977 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:16.072165 9977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:16.072342 9977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.5
I20240503 08:14:16.082296 9977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.123.5
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.5
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:16.083592 9977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:16.085178 9977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:16.096501 9984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.096938 9986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.097360 9983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:16.097815 9977 server_base.cc:1034] running on GCE node
I20240503 08:14:16.098338 9977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:16.099243 9977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:16.100430 9977 hybrid_clock.cc:648] HybridClock initialized: now 1714724056100387 us; error 47 us; skew 500 ppm
I20240503 08:14:16.100705 9977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:16.102772 9977 webserver.cc:460] Webserver started at http://127.24.123.5:45485/ using document root <none> and password file <none>
I20240503 08:14:16.103473 9977 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:16.103583 9977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:16.103879 9977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:16.106343 9977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a1402a75453d4ccf992606dcce77dc51"
format_stamp: "Formatted at 2024-05-03 08:14:16 on dist-test-slave-gkrw"
I20240503 08:14:16.107149 9977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a1402a75453d4ccf992606dcce77dc51"
format_stamp: "Formatted at 2024-05-03 08:14:16 on dist-test-slave-gkrw"
I20240503 08:14:16.111786 9977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:14:16.114898 9992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.116097 9977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:14:16.116241 9977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a1402a75453d4ccf992606dcce77dc51"
format_stamp: "Formatted at 2024-05-03 08:14:16 on dist-test-slave-gkrw"
I20240503 08:14:16.116410 9977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:16.138516 9977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:16.139343 9977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:16.139590 9977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:16.140506 9977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:16.142014 9977 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:16.142117 9977 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.142222 9977 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:16.142293 9977 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.170809 9977 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.5:43643
I20240503 08:14:16.170855 10104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.5:43643 every 8 connection(s)
I20240503 08:14:16.172062 9977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:14:16.175360 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 9977
I20240503 08:14:16.175511 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:14:16.178586 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.6:0
--local_ip_for_outbound_sockets=127.24.123.6
--webserver_interface=127.24.123.6
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:16.184970 10105 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:16.185365 10105 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.186267 10105 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:16.228544 9394 ts_manager.cc:194] Registered new tserver with Master: a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:16.229450 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.5:37645
I20240503 08:14:16.336383 9843 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
W20240503 08:14:16.502964 10108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:16.503278 10108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:16.503377 10108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:16.510700 10108 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:16.510820 10108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:16.510986 10108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.6
I20240503 08:14:16.519920 10108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.123.6
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.6
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:16.521195 10108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:16.522842 10108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:16.534528 10115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.534947 10117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.535468 10114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:16.536062 10108 server_base.cc:1034] running on GCE node
I20240503 08:14:16.536617 10108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:16.537508 10108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:16.538733 10108 hybrid_clock.cc:648] HybridClock initialized: now 1714724056538692 us; error 35 us; skew 500 ppm
I20240503 08:14:16.539077 10108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:16.541200 10108 webserver.cc:460] Webserver started at http://127.24.123.6:45345/ using document root <none> and password file <none>
I20240503 08:14:16.541878 10108 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:16.541983 10108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:16.542273 10108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:16.544756 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "fa60e870de844325bad4b9185eaee468"
format_stamp: "Formatted at 2024-05-03 08:14:16 on dist-test-slave-gkrw"
I20240503 08:14:16.545478 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "fa60e870de844325bad4b9185eaee468"
format_stamp: "Formatted at 2024-05-03 08:14:16 on dist-test-slave-gkrw"
I20240503 08:14:16.550288 10108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 08:14:16.553545 10123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.554767 10108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:16.554939 10108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fa60e870de844325bad4b9185eaee468"
format_stamp: "Formatted at 2024-05-03 08:14:16 on dist-test-slave-gkrw"
I20240503 08:14:16.555114 10108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:16.586319 10108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:16.587203 10108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:16.587458 10108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:16.588382 10108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:16.589787 10108 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:16.589876 10108 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.589978 10108 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:16.590046 10108 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.617547 10108 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.6:37175
I20240503 08:14:16.617609 10235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.6:37175 every 8 connection(s)
I20240503 08:14:16.618803 10108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:14:16.626112 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 10108
I20240503 08:14:16.626284 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:14:16.629738 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.7:0
--local_ip_for_outbound_sockets=127.24.123.7
--webserver_interface=127.24.123.7
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:16.631340 10236 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:16.631609 10236 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.632369 10236 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:16.673838 9394 ts_manager.cc:194] Registered new tserver with Master: fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:16.675086 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.6:52425
I20240503 08:14:16.790616 9974 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
W20240503 08:14:16.953843 10239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:16.954109 10239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:16.954232 10239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:16.961473 10239 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:16.961586 10239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:16.961755 10239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.7
I20240503 08:14:16.970660 10239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.123.7
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.7
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:16.972013 10239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:16.973675 10239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:16.989308 10245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:16.990195 10239 server_base.cc:1034] running on GCE node
W20240503 08:14:16.990496 10246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.990547 10248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:16.991235 10239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:16.992170 10239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:16.993366 10239 hybrid_clock.cc:648] HybridClock initialized: now 1714724056993302 us; error 57 us; skew 500 ppm
I20240503 08:14:16.993700 10239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:16.995860 10239 webserver.cc:460] Webserver started at http://127.24.123.7:46847/ using document root <none> and password file <none>
I20240503 08:14:16.996533 10239 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:16.996639 10239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:16.996933 10239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:16.999347 10239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "0cb18b8b10f944e39829faf16e061fd6"
format_stamp: "Formatted at 2024-05-03 08:14:16 on dist-test-slave-gkrw"
I20240503 08:14:17.000061 10239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "0cb18b8b10f944e39829faf16e061fd6"
format_stamp: "Formatted at 2024-05-03 08:14:16 on dist-test-slave-gkrw"
I20240503 08:14:17.004742 10239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:14:17.007917 10254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:17.009295 10239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:17.009438 10239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "0cb18b8b10f944e39829faf16e061fd6"
format_stamp: "Formatted at 2024-05-03 08:14:16 on dist-test-slave-gkrw"
I20240503 08:14:17.009616 10239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:17.030532 10239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:17.031419 10239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:17.031664 10239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:17.032641 10239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:17.034098 10239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:17.034188 10239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:17.034288 10239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:17.034356 10239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:17.065093 10239 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.7:42473
I20240503 08:14:17.065093 10366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.7:42473 every 8 connection(s)
I20240503 08:14:17.066574 10239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:14:17.076843 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 10239
I20240503 08:14:17.077024 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:14:17.081120 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.8:0
--local_ip_for_outbound_sockets=127.24.123.8
--webserver_interface=127.24.123.8
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:17.083389 10367 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:17.083709 10367 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:17.084678 10367 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:17.195020 9394 ts_manager.cc:194] Registered new tserver with Master: 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:17.196285 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.7:51125
I20240503 08:14:17.232051 10105 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
W20240503 08:14:17.406353 10370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:17.406611 10370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:17.406739 10370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:17.413928 10370 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:17.414062 10370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:17.414191 10370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.8
I20240503 08:14:17.422864 10370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.123.8
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.8
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:17.424053 10370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:17.425668 10370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:17.437034 10376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:17.437445 10377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:17.437790 10379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:17.438604 10370 server_base.cc:1034] running on GCE node
I20240503 08:14:17.439338 10370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:17.440279 10370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:17.441465 10370 hybrid_clock.cc:648] HybridClock initialized: now 1714724057441415 us; error 41 us; skew 500 ppm
I20240503 08:14:17.441751 10370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:17.443826 10370 webserver.cc:460] Webserver started at http://127.24.123.8:41085/ using document root <none> and password file <none>
I20240503 08:14:17.444496 10370 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:17.444599 10370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:17.444893 10370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:17.447283 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "ca5fa388d5b84a0abac1c05275155dcf"
format_stamp: "Formatted at 2024-05-03 08:14:17 on dist-test-slave-gkrw"
I20240503 08:14:17.447978 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "ca5fa388d5b84a0abac1c05275155dcf"
format_stamp: "Formatted at 2024-05-03 08:14:17 on dist-test-slave-gkrw"
I20240503 08:14:17.452715 10370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:14:17.455911 10385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:17.457451 10370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:14:17.457595 10370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ca5fa388d5b84a0abac1c05275155dcf"
format_stamp: "Formatted at 2024-05-03 08:14:17 on dist-test-slave-gkrw"
I20240503 08:14:17.457777 10370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:17.492246 10370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:17.493129 10370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:17.493357 10370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:17.494294 10370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:17.495739 10370 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:17.495831 10370 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:17.495944 10370 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:17.496018 10370 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:17.524350 10370 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.8:43883
I20240503 08:14:17.524403 10497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.8:43883 every 8 connection(s)
I20240503 08:14:17.525600 10370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:14:17.528111 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 10370
I20240503 08:14:17.528306 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:14:17.531589 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.9:0
--local_ip_for_outbound_sockets=127.24.123.9
--webserver_interface=127.24.123.9
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:17.537797 10498 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:17.538069 10498 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:17.538797 10498 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:17.580142 9394 ts_manager.cc:194] Registered new tserver with Master: ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883)
I20240503 08:14:17.581095 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.8:47353
I20240503 08:14:17.678135 10236 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
W20240503 08:14:17.852885 10501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:17.853140 10501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:17.853243 10501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:17.860493 10501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:17.860602 10501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:17.860757 10501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.9
I20240503 08:14:17.869638 10501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.9:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.24.123.9
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.9
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:17.870898 10501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:17.872555 10501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:17.883994 10507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:17.884521 10510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:17.884915 10508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:17.885360 10501 server_base.cc:1034] running on GCE node
I20240503 08:14:17.885895 10501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:17.886739 10501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:17.887926 10501 hybrid_clock.cc:648] HybridClock initialized: now 1714724057887880 us; error 42 us; skew 500 ppm
I20240503 08:14:17.888216 10501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:17.890311 10501 webserver.cc:460] Webserver started at http://127.24.123.9:43553/ using document root <none> and password file <none>
I20240503 08:14:17.891050 10501 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:17.891160 10501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:17.891436 10501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:17.895879 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "3a70e21b51a142bd8f7dc93fa58742fb"
format_stamp: "Formatted at 2024-05-03 08:14:17 on dist-test-slave-gkrw"
I20240503 08:14:17.896587 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "3a70e21b51a142bd8f7dc93fa58742fb"
format_stamp: "Formatted at 2024-05-03 08:14:17 on dist-test-slave-gkrw"
I20240503 08:14:17.901355 10501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:14:17.904507 10516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:17.905719 10501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:17.905858 10501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "3a70e21b51a142bd8f7dc93fa58742fb"
format_stamp: "Formatted at 2024-05-03 08:14:17 on dist-test-slave-gkrw"
I20240503 08:14:17.906036 10501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:17.928809 10501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:17.929608 10501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:17.929864 10501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:17.930804 10501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:17.932281 10501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:17.932379 10501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:17.932497 10501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:17.932564 10501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:17.961169 10501 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.9:44619
I20240503 08:14:17.961220 10628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.9:44619 every 8 connection(s)
I20240503 08:14:17.962368 10501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 08:14:17.967461 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 10501
I20240503 08:14:17.967597 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 08:14:17.970543 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.10:0
--local_ip_for_outbound_sockets=127.24.123.10
--webserver_interface=127.24.123.10
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:17.975739 10629 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:17.976032 10629 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:17.976832 10629 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:18.018770 9394 ts_manager.cc:194] Registered new tserver with Master: 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619)
I20240503 08:14:18.019724 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.9:50029
I20240503 08:14:18.198997 10367 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
W20240503 08:14:18.289047 10632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:18.289319 10632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:18.289443 10632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:18.296996 10632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:18.297120 10632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:18.297263 10632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.10
I20240503 08:14:18.305896 10632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.10:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.24.123.10
--webserver_port=0
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.10
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:18.307153 10632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:18.308782 10632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:18.320080 10641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:18.320453 10638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:18.320972 10639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:18.321529 10632 server_base.cc:1034] running on GCE node
I20240503 08:14:18.322075 10632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:18.322952 10632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:18.324137 10632 hybrid_clock.cc:648] HybridClock initialized: now 1714724058324094 us; error 42 us; skew 500 ppm
I20240503 08:14:18.324419 10632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:18.326406 10632 webserver.cc:460] Webserver started at http://127.24.123.10:39305/ using document root <none> and password file <none>
I20240503 08:14:18.327216 10632 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:18.327322 10632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:18.327600 10632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:18.329919 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data/instance:
uuid: "925c339b047b4cf5bae840c5a357d3de"
format_stamp: "Formatted at 2024-05-03 08:14:18 on dist-test-slave-gkrw"
I20240503 08:14:18.330610 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal/instance:
uuid: "925c339b047b4cf5bae840c5a357d3de"
format_stamp: "Formatted at 2024-05-03 08:14:18 on dist-test-slave-gkrw"
I20240503 08:14:18.335196 10632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:14:18.338137 10647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:18.339412 10632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:14:18.339541 10632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "925c339b047b4cf5bae840c5a357d3de"
format_stamp: "Formatted at 2024-05-03 08:14:18 on dist-test-slave-gkrw"
I20240503 08:14:18.339713 10632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:18.361035 10632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:18.361805 10632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:18.362043 10632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:18.362974 10632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:18.364380 10632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:18.364470 10632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:18.364569 10632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:18.364640 10632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:18.397385 10632 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.10:33439
I20240503 08:14:18.397435 10759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.10:33439 every 8 connection(s)
I20240503 08:14:18.398612 10632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 08:14:18.400372 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 10632
I20240503 08:14:18.400573 25068 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal/instance
I20240503 08:14:18.411028 10760 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:18.411296 10760 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:18.412010 10760 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:18.453122 9394 ts_manager.cc:194] Registered new tserver with Master: 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439)
I20240503 08:14:18.454030 9394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.10:35761
I20240503 08:14:18.457891 25068 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 08:14:18.475045 25068 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 08:14:18.475175 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 10501
I20240503 08:14:18.482573 10624 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:18.561017 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 10501
I20240503 08:14:18.574100 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 10632
I20240503 08:14:18.581853 10755 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:18.584314 10498 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
I20240503 08:14:18.661686 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 10632
I20240503 08:14:18.674150 25068 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEUWrla/build/asan/bin/kudu pid 10370
I20240503 08:14:18.682768 10493 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:14:18.761404 25068 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEUWrla/build/asan/bin/kudu with pid 10370
I20240503 08:14:22.526403 9394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55074:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:22.533087 9394 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:14:22.589646 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 34be73cc40264224ad798d7518c45ed1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.589632 10039 tablet_service.cc:1467] Processing CreateTablet for tablet 34be73cc40264224ad798d7518c45ed1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.591182 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be73cc40264224ad798d7518c45ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.591195 10039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be73cc40264224ad798d7518c45ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.590122 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2867608f0a4d69a3e169a59e358d10 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.593308 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2867608f0a4d69a3e169a59e358d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.599013 10038 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2867608f0a4d69a3e169a59e358d10 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.599501 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2867608f0a4d69a3e169a59e358d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.599819 10170 tablet_service.cc:1467] Processing CreateTablet for tablet 34be73cc40264224ad798d7518c45ed1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.600515 10040 tablet_service.cc:1467] Processing CreateTablet for tablet 014f053652224d60ac8ef85064f0cabe (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.601040 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014f053652224d60ac8ef85064f0cabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.601249 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be73cc40264224ad798d7518c45ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.602901 10171 tablet_service.cc:1467] Processing CreateTablet for tablet 014f053652224d60ac8ef85064f0cabe (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.603327 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014f053652224d60ac8ef85064f0cabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.606976 9647 tablet_service.cc:1467] Processing CreateTablet for tablet 014f053652224d60ac8ef85064f0cabe (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.600235 10169 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2867608f0a4d69a3e169a59e358d10 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.607889 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2867608f0a4d69a3e169a59e358d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.610920 10792 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:22.611327 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014f053652224d60ac8ef85064f0cabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.614356 10792 tablet_bootstrap.cc:654] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.615624 10792 log.cc:824] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:22.615928 10793 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:22.619547 10793 tablet_bootstrap.cc:654] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.621050 10793 log.cc:824] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:22.623407 10792 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:22.623713 10792 ts_tablet_manager.cc:1367] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20240503 08:14:22.630745 10793 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:22.631158 10793 ts_tablet_manager.cc:1367] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20240503 08:14:22.633283 10792 raft_consensus.cc:348] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.633961 10792 raft_consensus.cc:374] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.634114 10792 raft_consensus.cc:729] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.634910 10792 consensus_queue.cc:260] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.638056 10796 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:22.642700 9776 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2867608f0a4d69a3e169a59e358d10 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.642293 9777 tablet_service.cc:1467] Processing CreateTablet for tablet 34be73cc40264224ad798d7518c45ed1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.642758 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 34be73cc40264224ad798d7518c45ed1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.643795 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2867608f0a4d69a3e169a59e358d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.644188 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be73cc40264224ad798d7518c45ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.644234 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be73cc40264224ad798d7518c45ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.643805 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 34be73cc40264224ad798d7518c45ed1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.645062 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be73cc40264224ad798d7518c45ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.645172 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 014f053652224d60ac8ef85064f0cabe (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.645628 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014f053652224d60ac8ef85064f0cabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.646576 9907 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2867608f0a4d69a3e169a59e358d10 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.647038 9907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2867608f0a4d69a3e169a59e358d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.648289 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 014f053652224d60ac8ef85064f0cabe (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.648841 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014f053652224d60ac8ef85064f0cabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.649974 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2867608f0a4d69a3e169a59e358d10 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.650445 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2867608f0a4d69a3e169a59e358d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.655241 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 014f053652224d60ac8ef85064f0cabe (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.655843 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014f053652224d60ac8ef85064f0cabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.665113 10793 raft_consensus.cc:348] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.673125 10793 raft_consensus.cc:374] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.673261 10793 raft_consensus.cc:729] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.674428 10796 tablet_bootstrap.cc:654] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.677651 10800 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:22.678503 10793 consensus_queue.cc:260] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.680662 10796 log.cc:824] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:22.670826 10799 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:22.682921 10801 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:22.686529 10799 tablet_bootstrap.cc:654] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.686530 10801 tablet_bootstrap.cc:654] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.687048 10793 ts_tablet_manager.cc:1398] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.056s user 0.014s sys 0.000s
I20240503 08:14:22.687459 10793 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:22.687858 10799 log.cc:824] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:22.688293 10801 log.cc:824] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:22.689043 10793 tablet_bootstrap.cc:654] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.689922 10800 tablet_bootstrap.cc:654] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.691152 10796 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:22.691715 10800 log.cc:824] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:22.693464 10792 ts_tablet_manager.cc:1398] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.070s user 0.020s sys 0.022s
I20240503 08:14:22.693878 10792 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:22.695778 10792 tablet_bootstrap.cc:654] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.695945 10796 ts_tablet_manager.cc:1367] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.003s
I20240503 08:14:22.697304 10300 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2867608f0a4d69a3e169a59e358d10 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.698029 10793 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:22.698189 10793 ts_tablet_manager.cc:1367] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240503 08:14:22.699146 10793 raft_consensus.cc:348] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.699425 10793 raft_consensus.cc:374] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.699524 10793 raft_consensus.cc:729] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.699672 10801 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:22.699954 10801 ts_tablet_manager.cc:1367] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20240503 08:14:22.699852 10793 consensus_queue.cc:260] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.700757 10301 tablet_service.cc:1467] Processing CreateTablet for tablet 34be73cc40264224ad798d7518c45ed1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.701261 10793 ts_tablet_manager.cc:1398] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:22.701563 10793 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:22.702613 10302 tablet_service.cc:1467] Processing CreateTablet for tablet 014f053652224d60ac8ef85064f0cabe (DEFAULT_TABLE table=rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:22.698860 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2867608f0a4d69a3e169a59e358d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.704640 10793 tablet_bootstrap.cc:654] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.705528 10796 raft_consensus.cc:348] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.706099 10796 raft_consensus.cc:374] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.706228 10796 raft_consensus.cc:729] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.706372 10792 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:22.706538 10792 ts_tablet_manager.cc:1367] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 08:14:22.707007 10301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be73cc40264224ad798d7518c45ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.707242 10796 consensus_queue.cc:260] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.707453 10792 raft_consensus.cc:348] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.707722 10792 raft_consensus.cc:374] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.707809 10792 raft_consensus.cc:729] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.708143 10792 consensus_queue.cc:260] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.708312 10801 raft_consensus.cc:348] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.708786 10801 raft_consensus.cc:374] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.708884 10801 raft_consensus.cc:729] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.708931 10792 ts_tablet_manager.cc:1398] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:14:22.709775 10792 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:22.709573 10801 consensus_queue.cc:260] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.711449 10792 tablet_bootstrap.cc:654] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.712000 10302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014f053652224d60ac8ef85064f0cabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:22.714938 10793 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:22.715122 10793 ts_tablet_manager.cc:1367] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240503 08:14:22.715859 10793 raft_consensus.cc:348] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.717043 10793 raft_consensus.cc:374] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.717130 10793 raft_consensus.cc:729] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.717430 10793 consensus_queue.cc:260] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.718940 10793 ts_tablet_manager.cc:1398] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:22.719213 10796 ts_tablet_manager.cc:1398] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.023s user 0.007s sys 0.010s
I20240503 08:14:22.719697 10801 ts_tablet_manager.cc:1398] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.020s user 0.004s sys 0.014s
I20240503 08:14:22.719738 10796 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:22.721347 10792 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:22.721395 10796 tablet_bootstrap.cc:654] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.721504 10792 ts_tablet_manager.cc:1367] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:14:22.722158 10792 raft_consensus.cc:348] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.722380 10792 raft_consensus.cc:374] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.722445 10792 raft_consensus.cc:729] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.722695 10792 consensus_queue.cc:260] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.723196 10792 ts_tablet_manager.cc:1398] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:22.724701 10801 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:22.726308 10801 tablet_bootstrap.cc:654] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.733345 10796 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:22.733513 10796 ts_tablet_manager.cc:1367] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.008s
I20240503 08:14:22.734409 10796 raft_consensus.cc:348] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.734684 10796 raft_consensus.cc:374] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.734771 10796 raft_consensus.cc:729] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.735116 10796 consensus_queue.cc:260] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.735774 10796 ts_tablet_manager.cc:1398] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:14:22.736065 10796 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:22.737673 10796 tablet_bootstrap.cc:654] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.739822 10801 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:22.739928 10815 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:22.739992 10801 ts_tablet_manager.cc:1367] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20240503 08:14:22.740998 10801 raft_consensus.cc:348] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.741271 10801 raft_consensus.cc:374] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.741353 10801 raft_consensus.cc:729] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.741587 10801 consensus_queue.cc:260] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.742226 10801 ts_tablet_manager.cc:1398] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:22.742556 10801 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:22.744330 10801 tablet_bootstrap.cc:654] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.747714 10796 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:22.747911 10796 ts_tablet_manager.cc:1367] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240503 08:14:22.750048 10815 tablet_bootstrap.cc:654] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.751149 10815 log.cc:824] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:22.751889 10796 raft_consensus.cc:348] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.752133 10796 raft_consensus.cc:374] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.752199 10796 raft_consensus.cc:729] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.752472 10796 consensus_queue.cc:260] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.753049 10796 ts_tablet_manager.cc:1398] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:14:22.755954 10815 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:22.756311 10815 ts_tablet_manager.cc:1367] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20240503 08:14:22.761916 10799 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:22.762236 10799 ts_tablet_manager.cc:1367] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.092s user 0.000s sys 0.007s
I20240503 08:14:22.763374 10801 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:22.763545 10801 ts_tablet_manager.cc:1367] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240503 08:14:22.763391 10815 raft_consensus.cc:348] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.763816 10815 raft_consensus.cc:374] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.763909 10815 raft_consensus.cc:729] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.764228 10801 raft_consensus.cc:348] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.764525 10801 raft_consensus.cc:374] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.764600 10801 raft_consensus.cc:729] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.764636 10815 consensus_queue.cc:260] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.764869 10801 consensus_queue.cc:260] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.765465 10801 ts_tablet_manager.cc:1398] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:22.768874 10815 ts_tablet_manager.cc:1398] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20240503 08:14:22.769302 10815 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:22.771050 10815 tablet_bootstrap.cc:654] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.775696 10815 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:22.775843 10815 ts_tablet_manager.cc:1367] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240503 08:14:22.776732 10815 raft_consensus.cc:348] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.777004 10815 raft_consensus.cc:374] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.777091 10815 raft_consensus.cc:729] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.777330 10815 consensus_queue.cc:260] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.777880 10815 ts_tablet_manager.cc:1398] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:14:22.778215 10815 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:22.779748 10815 tablet_bootstrap.cc:654] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.779579 10799 raft_consensus.cc:348] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.780086 10799 raft_consensus.cc:374] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.780176 10799 raft_consensus.cc:729] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.780905 10799 consensus_queue.cc:260] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
W20240503 08:14:22.781862 9844 tablet.cc:2369] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:22.784924 10815 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:22.785048 10800 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:22.785095 10815 ts_tablet_manager.cc:1367] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240503 08:14:22.785347 10800 ts_tablet_manager.cc:1367] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.108s user 0.008s sys 0.003s
I20240503 08:14:22.785785 10815 raft_consensus.cc:348] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.786109 10815 raft_consensus.cc:374] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.786176 10815 raft_consensus.cc:729] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.786422 10815 consensus_queue.cc:260] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.787019 10815 ts_tablet_manager.cc:1398] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:14:22.791694 10800 raft_consensus.cc:348] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.792110 10799 ts_tablet_manager.cc:1398] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.030s user 0.010s sys 0.003s
I20240503 08:14:22.792289 10800 raft_consensus.cc:374] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.792389 10800 raft_consensus.cc:729] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.792564 10799 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:22.793212 10800 consensus_queue.cc:260] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.794641 10799 tablet_bootstrap.cc:654] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.796667 10799 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:22.796811 10799 ts_tablet_manager.cc:1367] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 08:14:22.796909 10800 ts_tablet_manager.cc:1398] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20240503 08:14:22.797257 10800 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:22.797689 10799 raft_consensus.cc:348] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.797948 10799 raft_consensus.cc:374] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.798036 10799 raft_consensus.cc:729] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.798267 10799 consensus_queue.cc:260] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.798928 10800 tablet_bootstrap.cc:654] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.799717 10799 ts_tablet_manager.cc:1398] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:14:22.800051 10799 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:22.801607 10799 tablet_bootstrap.cc:654] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.802834 10811 raft_consensus.cc:482] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:22.803177 10811 raft_consensus.cc:504] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.806849 10800 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:22.807018 10800 ts_tablet_manager.cc:1367] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20240503 08:14:22.807904 10800 raft_consensus.cc:348] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.808166 10800 raft_consensus.cc:374] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.808256 10800 raft_consensus.cc:729] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.808549 10800 consensus_queue.cc:260] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.809083 10800 ts_tablet_manager.cc:1398] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:22.809353 10800 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:22.810948 10800 tablet_bootstrap.cc:654] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:22.813205 10799 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:22.813505 10799 ts_tablet_manager.cc:1367] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240503 08:14:22.814368 10799 raft_consensus.cc:348] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.814843 10799 raft_consensus.cc:374] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.815007 10799 raft_consensus.cc:729] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.815307 10799 consensus_queue.cc:260] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.816030 10799 ts_tablet_manager.cc:1398] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:22.819703 10800 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:22.819904 10800 ts_tablet_manager.cc:1367] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:14:22.820722 10800 raft_consensus.cc:348] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.821022 10800 raft_consensus.cc:374] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:22.821098 10800 raft_consensus.cc:729] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:22.821388 10800 consensus_queue.cc:260] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
W20240503 08:14:22.821702 10368 tablet.cc:2369] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:22.822060 10800 ts_tablet_manager.cc:1398] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:14:22.825141 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:22.825662 10060 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.827244 10802 raft_consensus.cc:482] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:22.827503 10802 raft_consensus.cc:504] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.828019 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:22.828410 10191 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.830293 10802 leader_election.cc:288] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
W20240503 08:14:22.830770 9713 tablet.cc:2369] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:22.831512 10821 raft_consensus.cc:482] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:22.831781 10821 raft_consensus.cc:504] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.834583 10821 leader_election.cc:288] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:22.849202 10811 leader_election.cc:288] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
W20240503 08:14:22.874334 10237 tablet.cc:2369] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:22.879345 10811 raft_consensus.cc:482] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:22.879850 10811 raft_consensus.cc:504] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.881100 10811 leader_election.cc:288] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:22.890401 10823 raft_consensus.cc:482] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:22.890686 10823 raft_consensus.cc:504] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.891413 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:22.891644 10060 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.892298 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:22.892879 9666 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.894207 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:22.894480 9667 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.894968 9733 leader_election.cc:302] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:22.895478 10811 raft_consensus.cc:2781] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:22.895612 10811 raft_consensus.cc:482] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:22.895736 10811 raft_consensus.cc:3037] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.901008 10823 leader_election.cc:288] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
W20240503 08:14:22.905287 9581 tablet.cc:2369] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:22.908031 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:22.908284 9667 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:22.916951 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:22.917434 9929 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.920341 10811 raft_consensus.cc:504] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:22.922485 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:22.922719 10060 raft_consensus.cc:3037] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.925647 10060 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.926661 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:22.926932 9667 raft_consensus.cc:3037] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.926972 10811 leader_election.cc:288] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Requested vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:22.928035 10811 raft_consensus.cc:482] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 08:14:22.928114 10106 tablet.cc:2369] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:22.929672 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:22.929914 10060 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:22.929945 9667 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.930184 10811 raft_consensus.cc:504] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.933190 10811 leader_election.cc:288] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:22.933631 10322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:22.934207 10322 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.934664 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:22.934945 10321 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.935232 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:22.935473 10320 raft_consensus.cc:3037] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.938354 10320 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.938669 10319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:22.938935 10319 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.940989 9731 leader_election.cc:302] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:22.942013 10811 raft_consensus.cc:2781] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:22.942114 10811 raft_consensus.cc:482] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:22.943761 10811 raft_consensus.cc:3037] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.946518 10811 raft_consensus.cc:504] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.947868 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:22.948204 9667 raft_consensus.cc:2379] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06f1b17731834350a289de93abab8542 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:22.949354 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:22.949601 9929 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.950489 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:22.950687 9929 raft_consensus.cc:3037] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.953817 9929 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.954268 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:22.954502 9929 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.954847 9734 leader_election.cc:302] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51; no voters:
I20240503 08:14:22.959143 10852 raft_consensus.cc:2781] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:22.964444 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:22.964491 10319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:22.964669 10319 raft_consensus.cc:3037] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.966168 9929 raft_consensus.cc:3037] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.967499 10319 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.968242 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:22.968644 9535 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 0.
I20240503 08:14:22.968843 9929 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.976657 10319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:22.977054 10319 raft_consensus.cc:2379] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06f1b17731834350a289de93abab8542 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:22.978271 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:22.978490 9535 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:22.979224 10124 leader_election.cc:302] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:22.979668 10802 raft_consensus.cc:2781] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:22.979804 10802 raft_consensus.cc:482] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:22.979892 10802 raft_consensus.cc:3037] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.981216 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:22.981406 9535 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.981885 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:22.982064 9535 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.982446 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:22.982617 9535 raft_consensus.cc:3037] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.982771 10802 raft_consensus.cc:504] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:22.984207 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:22.984751 10802 leader_election.cc:288] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 election: Requested vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
W20240503 08:14:22.984779 9975 tablet.cc:2369] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:22.984874 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:22.984942 10811 leader_election.cc:288] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Requested vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:22.985100 10060 raft_consensus.cc:3037] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.986779 9535 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.986994 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:22.987322 9929 raft_consensus.cc:2379] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3d3436263d04c57b2d47cc9d766efa5 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:22.987854 10060 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:22.988258 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:22.988443 9535 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.988907 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:22.989223 9535 raft_consensus.cc:3037] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.989665 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:22.989900 10191 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:22.990131 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:22.990391 10190 raft_consensus.cc:3037] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.990327 10319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:22.990585 10319 raft_consensus.cc:2379] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3d3436263d04c57b2d47cc9d766efa5 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:22.990164 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:22.991159 9534 raft_consensus.cc:3037] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.993495 10190 raft_consensus.cc:2454] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.993916 9534 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:22.993847 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:22.994091 10189 raft_consensus.cc:3037] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.994856 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:22.995239 9665 raft_consensus.cc:3037] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.995441 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:22.995729 9928 raft_consensus.cc:3037] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:22.996729 10189 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.997073 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:22.997334 10191 raft_consensus.cc:2379] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de08e679a904ee48ba659861c730725 in current term 1: Already voted for candidate fa60e870de844325bad4b9185eaee468 in this term.
I20240503 08:14:22.997602 9535 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.997956 9665 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.997995 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:22.998451 9928 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:22.998817 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:22.999133 10060 raft_consensus.cc:2379] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de08e679a904ee48ba659861c730725 in current term 1: Already voted for candidate fa60e870de844325bad4b9185eaee468 in this term.
I20240503 08:14:22.999718 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2867608f0a4d69a3e169a59e358d10" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:22.999977 10059 raft_consensus.cc:3037] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:23.001973 10059 raft_consensus.cc:2454] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:22.995239 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:23.002475 10319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:23.002712 10319 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:23.002919 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:23.003144 10060 raft_consensus.cc:2379] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3d3436263d04c57b2d47cc9d766efa5 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:23.002528 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:23.003250 10124 leader_election.cc:302] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:23.003336 10320 raft_consensus.cc:3037] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:23.003362 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:23.003544 10802 raft_consensus.cc:2781] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:23.003481 9732 leader_election.cc:302] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:23.003639 10059 raft_consensus.cc:2379] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06f1b17731834350a289de93abab8542 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:23.004365 10857 raft_consensus.cc:2781] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:23.004530 10857 raft_consensus.cc:686] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Becoming Leader. State: Replica: 8de08e679a904ee48ba659861c730725, State: Running, Role: LEADER
I20240503 08:14:23.004971 10852 raft_consensus.cc:686] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Becoming Leader. State: Replica: 8de08e679a904ee48ba659861c730725, State: Running, Role: LEADER
I20240503 08:14:23.005075 9731 leader_election.cc:302] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5; no voters: fa60e870de844325bad4b9185eaee468
I20240503 08:14:23.005430 10811 raft_consensus.cc:2781] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:23.005563 10811 raft_consensus.cc:482] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:23.005652 10811 raft_consensus.cc:3037] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:23.006075 10320 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:23.007079 10852 consensus_queue.cc:237] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:23.008744 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:23.008427 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:23.009394 10857 consensus_queue.cc:237] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:23.010808 9667 raft_consensus.cc:3037] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:23.011797 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:23.012075 9665 raft_consensus.cc:2379] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3d3436263d04c57b2d47cc9d766efa5 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:23.012658 9470 leader_election.cc:302] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: b3d3436263d04c57b2d47cc9d766efa5; no voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1
I20240503 08:14:23.013151 10823 raft_consensus.cc:2726] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:23.013494 9667 raft_consensus.cc:2454] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:23.013530 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:23.013787 10191 raft_consensus.cc:2379] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06f1b17731834350a289de93abab8542 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:23.013952 9666 raft_consensus.cc:2379] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de08e679a904ee48ba659861c730725 in current term 1: Already voted for candidate fa60e870de844325bad4b9185eaee468 in this term.
I20240503 08:14:23.015373 9863 leader_election.cc:302] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 06f1b17731834350a289de93abab8542, b3d3436263d04c57b2d47cc9d766efa5; no voters: 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468
I20240503 08:14:23.015795 10821 raft_consensus.cc:2726] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:23.016939 10802 raft_consensus.cc:686] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Becoming Leader. State: Replica: fa60e870de844325bad4b9185eaee468, State: Running, Role: LEADER
I20240503 08:14:23.017180 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:23.017383 10191 raft_consensus.cc:2379] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3d3436263d04c57b2d47cc9d766efa5 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:23.017436 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:23.018247 10802 consensus_queue.cc:237] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:23.019935 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be73cc40264224ad798d7518c45ed1" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:23.020154 10811 raft_consensus.cc:504] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:23.020787 9797 raft_consensus.cc:2379] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate 8de08e679a904ee48ba659861c730725 in this term.
I20240503 08:14:23.021827 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:23.022063 9666 raft_consensus.cc:2379] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de08e679a904ee48ba659861c730725 in current term 1: Already voted for candidate fa60e870de844325bad4b9185eaee468 in this term.
I20240503 08:14:23.021929 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:23.022622 9535 raft_consensus.cc:2379] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de08e679a904ee48ba659861c730725 in current term 1: Already voted for candidate fa60e870de844325bad4b9185eaee468 in this term.
I20240503 08:14:23.024202 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:23.024222 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:23.024436 10320 raft_consensus.cc:2379] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de08e679a904ee48ba659861c730725 in current term 1: Already voted for candidate fa60e870de844325bad4b9185eaee468 in this term.
I20240503 08:14:23.024344 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:23.024612 10059 raft_consensus.cc:2379] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de08e679a904ee48ba659861c730725 in current term 1: Already voted for candidate fa60e870de844325bad4b9185eaee468 in this term.
I20240503 08:14:23.024885 9731 leader_election.cc:302] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 8de08e679a904ee48ba659861c730725; no voters: 0cb18b8b10f944e39829faf16e061fd6, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468
I20240503 08:14:23.024904 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014f053652224d60ac8ef85064f0cabe" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:23.025161 9929 raft_consensus.cc:2379] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de08e679a904ee48ba659861c730725 in current term 1: Already voted for candidate fa60e870de844325bad4b9185eaee468 in this term.
I20240503 08:14:23.025357 10811 leader_election.cc:288] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Requested vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:23.025602 10811 raft_consensus.cc:2726] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:14:23.037173 9394 catalog_manager.cc:5484] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de08e679a904ee48ba659861c730725 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:23.037122 9393 catalog_manager.cc:5484] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa60e870de844325bad4b9185eaee468 (127.24.123.6). New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:23.044093 9394 catalog_manager.cc:5484] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de08e679a904ee48ba659861c730725 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:23.399586 10857 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.424535 10857 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.450618 10857 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:23.460932 10864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:23.461223 10864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:23.469076 10864 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:14:23.504778 10802 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.509619 10802 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.510618 10857 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.533881 10802 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.542344 10802 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.551801 10802 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.553856 10857 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.560257 10857 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.563261 10802 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.598592 10857 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.604861 10916 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:14:23.614929 10902 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.624817 10857 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.626194 10857 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:23.634080 10857 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 518.925 ms
time per row: 0.259463 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:14:25.004333 9386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55074:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:25.005012 9386 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:14:25.013782 9647 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.014250 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.014899 10040 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.015137 9907 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.015307 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.015537 9907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.018462 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6758eb87b047d29a060fcbfb775903 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.018582 10169 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.018769 9514 tablet_service.cc:1467] Processing CreateTablet for tablet e11a6197ccdd4e8d9a203efae4a94988 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.019510 9777 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.019650 10300 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.019853 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.020045 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.021153 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6758eb87b047d29a060fcbfb775903 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.021520 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.023273 10302 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6758eb87b047d29a060fcbfb775903 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.023303 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6758eb87b047d29a060fcbfb775903 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.023649 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.023669 10302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.024920 9909 tablet_service.cc:1467] Processing CreateTablet for tablet e11a6197ccdd4e8d9a203efae4a94988 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.025283 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.027284 10301 tablet_service.cc:1467] Processing CreateTablet for tablet e11a6197ccdd4e8d9a203efae4a94988 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.027699 10301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.019460 9776 tablet_service.cc:1467] Processing CreateTablet for tablet e11a6197ccdd4e8d9a203efae4a94988 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.028198 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.031875 10171 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6758eb87b047d29a060fcbfb775903 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.032269 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.032964 10170 tablet_service.cc:1467] Processing CreateTablet for tablet e11a6197ccdd4e8d9a203efae4a94988 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.018862 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.035116 9645 tablet_service.cc:1467] Processing CreateTablet for tablet e11a6197ccdd4e8d9a203efae4a94988 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.035307 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6758eb87b047d29a060fcbfb775903 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.035501 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.035712 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.036139 10039 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6758eb87b047d29a060fcbfb775903 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.036520 10039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.019263 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.039362 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.039508 10038 tablet_service.cc:1467] Processing CreateTablet for tablet e11a6197ccdd4e8d9a203efae4a94988 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:25.039903 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.043026 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.043501 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.046029 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:25.057873 10976 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:25.058494 10977 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:25.062302 10976 tablet_bootstrap.cc:654] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.062480 10977 tablet_bootstrap.cc:654] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.062922 10979 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:25.063416 10978 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:25.064185 10975 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:25.065202 10977 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:25.065441 10977 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:14:25.065759 10978 tablet_bootstrap.cc:654] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.066314 10977 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.066704 10977 raft_consensus.cc:374] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.066831 10977 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.067126 10977 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.067827 10980 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:25.068964 10975 tablet_bootstrap.cc:654] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.069900 10979 tablet_bootstrap.cc:654] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.071518 10980 tablet_bootstrap.cc:654] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.074194 10977 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
I20240503 08:14:25.075479 10977 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:25.076126 10978 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:25.076346 10978 ts_tablet_manager.cc:1367] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 08:14:25.077116 10978 raft_consensus.cc:348] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.077451 10978 raft_consensus.cc:374] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.077467 10977 tablet_bootstrap.cc:654] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.077582 10978 raft_consensus.cc:729] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.077880 10978 consensus_queue.cc:260] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.079453 10979 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:25.079633 10979 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.005s
I20240503 08:14:25.080389 10975 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:25.080459 10979 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.080654 10975 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 08:14:25.080722 10979 raft_consensus.cc:374] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.080819 10979 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.081100 10979 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.081565 10975 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.081830 10975 raft_consensus.cc:374] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.081921 10975 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.081780 10974 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:25.082185 10975 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.084146 10974 tablet_bootstrap.cc:654] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.085107 10975 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:25.087440 10975 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:25.089222 10975 tablet_bootstrap.cc:654] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.090857 10979 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.011s user 0.000s sys 0.005s
I20240503 08:14:25.092000 10980 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:25.092221 10980 ts_tablet_manager.cc:1367] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20240503 08:14:25.093061 10980 raft_consensus.cc:348] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.093390 10980 raft_consensus.cc:374] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.093525 10980 raft_consensus.cc:729] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.093762 10978 ts_tablet_manager.cc:1398] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240503 08:14:25.094127 10979 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:25.094190 10978 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:25.094264 10980 consensus_queue.cc:260] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.096204 10979 tablet_bootstrap.cc:654] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.096772 10980 ts_tablet_manager.cc:1398] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:25.095911 10976 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:25.097128 10980 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:25.098555 10980 tablet_bootstrap.cc:654] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.099712 10978 tablet_bootstrap.cc:654] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.100214 10976 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.005s
I20240503 08:14:25.100422 10974 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:25.100636 10974 ts_tablet_manager.cc:1367] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 08:14:25.101450 10974 raft_consensus.cc:348] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.101828 10974 raft_consensus.cc:374] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.101949 10974 raft_consensus.cc:729] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.102509 10974 consensus_queue.cc:260] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.102916 10975 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:25.102721 10976 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.103152 10975 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240503 08:14:25.103178 10976 raft_consensus.cc:374] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.103374 10976 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.103706 10976 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.103968 10975 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.104331 10975 raft_consensus.cc:374] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.104441 10975 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.104750 10975 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.105296 10978 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:25.105512 10978 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:14:25.106242 10978 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.106554 10978 raft_consensus.cc:374] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.106667 10978 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.106992 10978 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.107601 10978 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:25.107949 10978 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:25.108119 10975 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:14:25.109361 10979 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:25.109545 10978 tablet_bootstrap.cc:654] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.110478 10979 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 08:14:25.110708 10975 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:25.111688 10979 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.112248 10977 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:25.112459 10977 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.027s
I20240503 08:14:25.112716 10975 tablet_bootstrap.cc:654] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.113340 10977 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.113646 10977 raft_consensus.cc:374] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.113723 10977 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.114046 10977 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.114457 10980 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:25.114612 10979 raft_consensus.cc:374] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.114897 10980 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
I20240503 08:14:25.115017 10979 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.115329 10977 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:14:25.115620 10977 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:25.116223 10980 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.115579 10979 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.117154 10979 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:14:25.117817 10979 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:25.117954 10980 raft_consensus.cc:374] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.118077 10980 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.118566 10977 tablet_bootstrap.cc:654] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.118830 10980 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.119297 10976 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240503 08:14:25.119648 10976 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:25.120313 10979 tablet_bootstrap.cc:654] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.121454 10980 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:14:25.121796 10980 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:25.122249 10974 ts_tablet_manager.cc:1398] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.021s user 0.002s sys 0.003s
I20240503 08:14:25.122633 10974 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:25.124939 10974 tablet_bootstrap.cc:654] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.127030 10976 tablet_bootstrap.cc:654] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.127476 10980 tablet_bootstrap.cc:654] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.128055 10975 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:25.128260 10975 ts_tablet_manager.cc:1367] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.002s
I20240503 08:14:25.130957 10975 raft_consensus.cc:348] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.131331 10975 raft_consensus.cc:374] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.131450 10975 raft_consensus.cc:729] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.131748 10975 consensus_queue.cc:260] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.134758 10975 ts_tablet_manager.cc:1398] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:14:25.137611 10980 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:25.138348 10980 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 08:14:25.138473 10979 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:25.138530 10978 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:25.138685 10978 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.008s
I20240503 08:14:25.138712 10979 ts_tablet_manager.cc:1367] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240503 08:14:25.139086 10976 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:25.139298 10976 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.005s
I20240503 08:14:25.139454 10980 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.139477 10978 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.139732 10978 raft_consensus.cc:374] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.139732 10980 raft_consensus.cc:374] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.139830 10980 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.139931 10978 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.139708 10979 raft_consensus.cc:348] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.140136 10979 raft_consensus.cc:374] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.140271 10979 raft_consensus.cc:729] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.140082 10980 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.140221 10976 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.140245 10978 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.140501 10976 raft_consensus.cc:374] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.140591 10976 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.140940 10978 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:25.141265 10979 consensus_queue.cc:260] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.141485 10980 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:25.141988 10977 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:25.142032 10979 ts_tablet_manager.cc:1398] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:25.142184 10977 ts_tablet_manager.cc:1367] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.013s
I20240503 08:14:25.142993 10977 raft_consensus.cc:348] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.143275 10977 raft_consensus.cc:374] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.143399 10977 raft_consensus.cc:729] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.143800 10974 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:25.143652 10977 consensus_queue.cc:260] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.144001 10974 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.005s
I20240503 08:14:25.144328 10977 ts_tablet_manager.cc:1398] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:25.140846 10976 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.146072 10976 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:14:25.146401 10976 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:25.147768 10974 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.148008 10976 tablet_bootstrap.cc:654] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.148084 10974 raft_consensus.cc:374] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.148166 10974 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.149250 10974 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.149745 10976 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:25.149834 10974 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:14:25.149889 10976 ts_tablet_manager.cc:1367] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:14:25.150188 10974 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:25.150480 10976 raft_consensus.cc:348] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.150694 10976 raft_consensus.cc:374] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.150806 10976 raft_consensus.cc:729] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.151033 10976 consensus_queue.cc:260] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.151558 10976 ts_tablet_manager.cc:1398] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:25.151937 10974 tablet_bootstrap.cc:654] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:25.153518 10974 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:25.153657 10974 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:14:25.154546 10974 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.154788 10974 raft_consensus.cc:374] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:25.154904 10974 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:25.155120 10974 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.155543 10974 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:25.164539 10990 raft_consensus.cc:482] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:25.164762 10990 raft_consensus.cc:504] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.167035 10990 leader_election.cc:288] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565)
I20240503 08:14:25.171316 10989 raft_consensus.cc:482] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:25.171627 10989 raft_consensus.cc:504] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.172945 10989 leader_election.cc:288] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:25.173444 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:25.173698 10191 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:25.173995 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:25.174247 9666 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:25.174693 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:25.174994 9797 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:25.175943 9468 leader_election.cc:302] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:25.176699 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:25.176946 10060 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:25.177482 10989 raft_consensus.cc:2781] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:25.177613 10989 raft_consensus.cc:482] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:25.177712 10989 raft_consensus.cc:3037] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.180024 10997 raft_consensus.cc:482] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:25.180223 10997 raft_consensus.cc:504] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.182729 10997 leader_election.cc:288] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:25.182893 10989 raft_consensus.cc:504] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.185117 10989 leader_election.cc:288] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Requested vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:25.185307 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:25.185545 10191 raft_consensus.cc:3037] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.186776 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:25.187036 9797 raft_consensus.cc:3037] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.188320 10191 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:25.189980 9797 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:25.190788 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:25.191023 9666 raft_consensus.cc:3037] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.193866 9666 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:25.194276 9470 leader_election.cc:302] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:25.199018 10989 raft_consensus.cc:2781] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:25.211962 10989 raft_consensus.cc:686] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Becoming Leader. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Running, Role: LEADER
I20240503 08:14:25.212637 10989 consensus_queue.cc:237] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.223265 9386 catalog_manager.cc:5484] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:25.228194 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:25.228461 10060 raft_consensus.cc:3037] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.228961 9927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:25.229180 9927 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:25.229511 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:25.229702 9929 raft_consensus.cc:3037] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.230011 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:25.231317 10060 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:25.232378 9929 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:25.237592 10322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:25.237833 10322 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:25.238425 10322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:25.238626 10322 raft_consensus.cc:3037] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.240000 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:25.240314 9797 raft_consensus.cc:2379] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:25.241914 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:25.242184 9929 raft_consensus.cc:2379] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:25.242589 10322 raft_consensus.cc:2454] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:25.244004 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:25.244275 10191 raft_consensus.cc:2379] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:25.244622 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:25.244791 10256 leader_election.cc:302] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6; no voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468
I20240503 08:14:25.244923 9666 raft_consensus.cc:2379] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:25.245193 10990 raft_consensus.cc:2726] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:25.255556 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:25.255895 9534 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:25.258492 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:25.258757 9797 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:25.259692 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:25.259917 10191 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:25.260373 9994 leader_election.cc:302] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:25.262533 10997 raft_consensus.cc:2781] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:25.262681 10997 raft_consensus.cc:482] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:25.262801 10997 raft_consensus.cc:3037] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.263720 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:25.263808 10322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:25.263934 9929 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:25.264029 10322 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:25.264654 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:25.264878 9666 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:25.265683 10997 raft_consensus.cc:504] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.266788 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:25.266851 10997 leader_election.cc:288] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Requested vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:25.267009 9666 raft_consensus.cc:3037] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.267660 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:25.267673 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:25.267844 9929 raft_consensus.cc:3037] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.267863 9797 raft_consensus.cc:3037] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.269062 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9fcac8b4e14bffab9fb50d2fe8660f" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:25.269295 10060 raft_consensus.cc:2379] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:25.269554 9666 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:25.270174 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:25.270375 9534 raft_consensus.cc:3037] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.270489 9797 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:25.270501 9929 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:25.270784 10322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:25.271021 10322 raft_consensus.cc:3037] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.270956 9993 leader_election.cc:302] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:25.271093 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6758eb87b047d29a060fcbfb775903" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:25.271265 10997 raft_consensus.cc:2781] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:25.271282 10191 raft_consensus.cc:3037] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.273483 9534 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:25.273969 10322 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:25.274083 10191 raft_consensus.cc:2454] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:25.275094 10997 raft_consensus.cc:686] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Becoming Leader. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Running, Role: LEADER
I20240503 08:14:25.275637 10997 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.280040 9386 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1402a75453d4ccf992606dcce77dc51 (127.24.123.5). New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:25.336359 10990 raft_consensus.cc:482] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:25.336572 10990 raft_consensus.cc:504] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.337558 10990 leader_election.cc:288] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:25.337759 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:25.338004 9666 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:25.337901 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:25.338168 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:25.338423 9929 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:25.338603 9797 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:25.338814 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:25.339071 10191 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:25.339078 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:25.339176 10255 leader_election.cc:302] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:25.339303 9534 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:25.339656 10990 raft_consensus.cc:2781] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:25.339601 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:25.339790 10990 raft_consensus.cc:482] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:25.339826 10060 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:25.339890 10990 raft_consensus.cc:3037] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.341894 10990 raft_consensus.cc:504] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.343015 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:25.343159 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:25.343186 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:25.343261 9534 raft_consensus.cc:3037] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.343230 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:25.343350 9666 raft_consensus.cc:3037] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.343366 10191 raft_consensus.cc:3037] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.343521 9797 raft_consensus.cc:3037] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.346012 10191 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:25.346040 9666 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:25.346046 9534 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:25.346257 9797 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:25.346450 10255 leader_election.cc:302] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:25.347256 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:25.347462 10990 leader_election.cc:288] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 election: Requested vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:25.347435 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a6197ccdd4e8d9a203efae4a94988" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:25.347548 11034 raft_consensus.cc:2781] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:25.347613 9929 raft_consensus.cc:3037] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.347658 10060 raft_consensus.cc:3037] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:25.347719 11034 raft_consensus.cc:686] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Running, Role: LEADER
I20240503 08:14:25.348253 11034 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:25.350261 10060 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:25.350261 9929 raft_consensus.cc:2454] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:25.352603 9386 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7). New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:25.622159 10989 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.623809 10989 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.680892 10997 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.687712 10997 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.703243 10997 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.710085 10997 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.710601 10997 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.715023 10997 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.729873 11034 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.749390 10989 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.782447 11034 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.796394 10989 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.802034 10989 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.805361 11034 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.806686 11024 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:25.817350 11069 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:25.817417 11036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:25.817844 11036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:14:25.819483 10990 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:25.832760 11036 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:25.843287 11069 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 365.2 ms
time per row: 0.1826 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:14:27.228027 9390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55074:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:27.228709 9390 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:14:27.237754 9513 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f0afdc01d4d03917b9040c2053426 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.237978 9646 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f0afdc01d4d03917b9040c2053426 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.238261 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f0afdc01d4d03917b9040c2053426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.238251 9776 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f0afdc01d4d03917b9040c2053426 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.238309 10038 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f0afdc01d4d03917b9040c2053426 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.238466 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f0afdc01d4d03917b9040c2053426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.238662 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f0afdc01d4d03917b9040c2053426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.238715 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f0afdc01d4d03917b9040c2053426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.244365 9907 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f0afdc01d4d03917b9040c2053426 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.244833 9907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f0afdc01d4d03917b9040c2053426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.252156 10169 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f0afdc01d4d03917b9040c2053426 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.252704 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f0afdc01d4d03917b9040c2053426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.253324 9909 tablet_service.cc:1467] Processing CreateTablet for tablet dd41cbc6c69441c1af5c36c382cfe366 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.253726 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41cbc6c69441c1af5c36c382cfe366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.256263 9647 tablet_service.cc:1467] Processing CreateTablet for tablet dd41cbc6c69441c1af5c36c382cfe366 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.256670 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41cbc6c69441c1af5c36c382cfe366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.257334 10302 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f0afdc01d4d03917b9040c2053426 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.257779 10302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f0afdc01d4d03917b9040c2053426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.261801 10040 tablet_service.cc:1467] Processing CreateTablet for tablet dd41cbc6c69441c1af5c36c382cfe366 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.262200 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41cbc6c69441c1af5c36c382cfe366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.262225 9515 tablet_service.cc:1467] Processing CreateTablet for tablet dd41cbc6c69441c1af5c36c382cfe366 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.262652 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41cbc6c69441c1af5c36c382cfe366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.267184 9777 tablet_service.cc:1467] Processing CreateTablet for tablet dd41cbc6c69441c1af5c36c382cfe366 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.267606 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41cbc6c69441c1af5c36c382cfe366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.270280 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 8506211351284b7a8e2a166b8083977a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.270280 10170 tablet_service.cc:1467] Processing CreateTablet for tablet dd41cbc6c69441c1af5c36c382cfe366 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.270694 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8506211351284b7a8e2a166b8083977a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.270732 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41cbc6c69441c1af5c36c382cfe366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.278982 10040 tablet_service.cc:1467] Processing CreateTablet for tablet 8506211351284b7a8e2a166b8083977a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.280318 10171 tablet_service.cc:1467] Processing CreateTablet for tablet 8506211351284b7a8e2a166b8083977a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.280810 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8506211351284b7a8e2a166b8083977a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.281539 9647 tablet_service.cc:1467] Processing CreateTablet for tablet 8506211351284b7a8e2a166b8083977a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.281948 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8506211351284b7a8e2a166b8083977a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.284667 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8506211351284b7a8e2a166b8083977a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.284612 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 8506211351284b7a8e2a166b8083977a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.285058 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8506211351284b7a8e2a166b8083977a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.287211 11152 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:27.287513 10300 tablet_service.cc:1467] Processing CreateTablet for tablet dd41cbc6c69441c1af5c36c382cfe366 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.287531 10301 tablet_service.cc:1467] Processing CreateTablet for tablet 8506211351284b7a8e2a166b8083977a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.287984 10301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8506211351284b7a8e2a166b8083977a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.288157 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41cbc6c69441c1af5c36c382cfe366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.292889 11152 tablet_bootstrap.cc:654] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.293578 11151 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:27.300253 11148 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:27.301985 11148 tablet_bootstrap.cc:654] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.295323 9777 tablet_service.cc:1467] Processing CreateTablet for tablet 8506211351284b7a8e2a166b8083977a (DEFAULT_TABLE table=rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:27.305886 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8506211351284b7a8e2a166b8083977a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:27.307299 11148 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:27.308084 11148 ts_tablet_manager.cc:1367] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:14:27.309134 11148 raft_consensus.cc:348] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.310361 11150 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:27.312624 11150 tablet_bootstrap.cc:654] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.312932 11148 raft_consensus.cc:374] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.313220 11148 raft_consensus.cc:729] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.313807 11148 consensus_queue.cc:260] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.319263 11148 ts_tablet_manager.cc:1398] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 08:14:27.323683 11150 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:27.326292 11148 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:27.327987 11151 tablet_bootstrap.cc:654] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.328176 11148 tablet_bootstrap.cc:654] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.329124 11150 ts_tablet_manager.cc:1367] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 08:14:27.329864 11150 raft_consensus.cc:348] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.330152 11150 raft_consensus.cc:374] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.330237 11150 raft_consensus.cc:729] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.331830 11150 consensus_queue.cc:260] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.333047 11150 ts_tablet_manager.cc:1398] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:27.335322 11161 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:27.336966 11161 tablet_bootstrap.cc:654] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.337932 11150 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:27.341929 11150 tablet_bootstrap.cc:654] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.343449 11153 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:27.345299 11153 tablet_bootstrap.cc:654] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.346375 11151 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:27.346588 11151 ts_tablet_manager.cc:1367] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.004s
I20240503 08:14:27.346921 11155 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:27.347393 11151 raft_consensus.cc:348] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.347841 11151 raft_consensus.cc:374] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.347965 11151 raft_consensus.cc:729] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.348253 11151 consensus_queue.cc:260] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.348994 11151 ts_tablet_manager.cc:1398] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:27.349958 11155 tablet_bootstrap.cc:654] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.351025 11151 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:27.351441 11153 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:27.351583 11153 ts_tablet_manager.cc:1367] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240503 08:14:27.352293 11153 raft_consensus.cc:348] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.352576 11153 raft_consensus.cc:374] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.352666 11153 raft_consensus.cc:729] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.352922 11153 consensus_queue.cc:260] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.354984 11151 tablet_bootstrap.cc:654] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.356038 11148 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:27.356316 11148 ts_tablet_manager.cc:1367] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20240503 08:14:27.357079 11148 raft_consensus.cc:348] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.357880 11150 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:27.357899 11152 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:27.358042 11152 ts_tablet_manager.cc:1367] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.084s user 0.021s sys 0.011s
I20240503 08:14:27.358109 11150 ts_tablet_manager.cc:1367] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240503 08:14:27.358814 11150 raft_consensus.cc:348] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.358827 11152 raft_consensus.cc:348] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.359040 11153 ts_tablet_manager.cc:1398] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20240503 08:14:27.359112 11152 raft_consensus.cc:374] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.359223 11148 raft_consensus.cc:374] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.359301 11152 raft_consensus.cc:729] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.359323 11153 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:27.359364 11150 raft_consensus.cc:374] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.359483 11148 raft_consensus.cc:729] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.359597 11150 raft_consensus.cc:729] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.360312 11148 consensus_queue.cc:260] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.360757 11150 consensus_queue.cc:260] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.361133 11153 tablet_bootstrap.cc:654] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.359613 11152 consensus_queue.cc:260] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.361594 11148 ts_tablet_manager.cc:1398] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:14:27.361884 11152 ts_tablet_manager.cc:1398] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:27.363711 11150 ts_tablet_manager.cc:1398] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:14:27.364065 11148 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:27.364373 11150 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:27.367334 11148 tablet_bootstrap.cc:654] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.367434 11161 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:27.367586 11161 ts_tablet_manager.cc:1367] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20240503 08:14:27.367843 11150 tablet_bootstrap.cc:654] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.368368 11161 raft_consensus.cc:348] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.368644 11161 raft_consensus.cc:374] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.368739 11161 raft_consensus.cc:729] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.369186 11155 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:27.369026 11161 consensus_queue.cc:260] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.370127 11161 ts_tablet_manager.cc:1398] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:27.370451 11161 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:27.362239 11152 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:27.372722 11152 tablet_bootstrap.cc:654] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.373934 11161 tablet_bootstrap.cc:654] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.375566 11155 ts_tablet_manager.cc:1367] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.010s
I20240503 08:14:27.376458 11155 raft_consensus.cc:348] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.376811 11155 raft_consensus.cc:374] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.376935 11155 raft_consensus.cc:729] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.377214 11155 consensus_queue.cc:260] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.378029 11155 ts_tablet_manager.cc:1398] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:27.378396 11155 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:27.381875 11151 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:27.382117 11151 ts_tablet_manager.cc:1367] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20240503 08:14:27.383090 11151 raft_consensus.cc:348] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.383318 11148 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:27.383422 11151 raft_consensus.cc:374] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.383548 11151 raft_consensus.cc:729] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.383744 11148 ts_tablet_manager.cc:1367] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.001s
I20240503 08:14:27.383922 11155 tablet_bootstrap.cc:654] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.383880 11151 consensus_queue.cc:260] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.384653 11148 raft_consensus.cc:348] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.385206 11148 raft_consensus.cc:374] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.385349 11148 raft_consensus.cc:729] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.385632 11148 consensus_queue.cc:260] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.386478 11148 ts_tablet_manager.cc:1398] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:14:27.387526 11150 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:27.387727 11150 ts_tablet_manager.cc:1367] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.009s
I20240503 08:14:27.388532 11150 raft_consensus.cc:348] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.388849 11150 raft_consensus.cc:374] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.388985 11150 raft_consensus.cc:729] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.389304 11150 consensus_queue.cc:260] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.390072 11150 ts_tablet_manager.cc:1398] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:27.391163 11161 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:27.391530 11152 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:27.391686 11151 ts_tablet_manager.cc:1398] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 08:14:27.391907 11161 ts_tablet_manager.cc:1367] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20240503 08:14:27.392606 11161 raft_consensus.cc:348] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.392894 11161 raft_consensus.cc:374] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.392971 11161 raft_consensus.cc:729] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.393213 11161 consensus_queue.cc:260] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.393798 11161 ts_tablet_manager.cc:1398] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:27.394186 11151 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:27.395812 11151 tablet_bootstrap.cc:654] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.396814 11152 ts_tablet_manager.cc:1367] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.001s
I20240503 08:14:27.397563 11152 raft_consensus.cc:348] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.397840 11152 raft_consensus.cc:374] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.397909 11152 raft_consensus.cc:729] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.398190 11152 consensus_queue.cc:260] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.398737 11152 ts_tablet_manager.cc:1398] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:14:27.399102 11152 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:27.400442 11153 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:27.400548 11152 tablet_bootstrap.cc:654] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.400913 11153 ts_tablet_manager.cc:1367] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.007s
I20240503 08:14:27.401646 11161 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:27.401788 11153 raft_consensus.cc:348] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.402827 11153 raft_consensus.cc:374] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.402962 11153 raft_consensus.cc:729] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.402969 11157 raft_consensus.cc:482] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:27.403175 11161 tablet_bootstrap.cc:654] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.403147 11157 raft_consensus.cc:504] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.403285 11153 consensus_queue.cc:260] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.403810 11153 ts_tablet_manager.cc:1398] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:14:27.404106 11153 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:27.405659 11153 tablet_bootstrap.cc:654] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.405632 11157 leader_election.cc:288] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:27.411908 11151 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:27.412076 11151 ts_tablet_manager.cc:1367] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20240503 08:14:27.412839 11151 raft_consensus.cc:348] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.413091 11151 raft_consensus.cc:374] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.413219 11151 raft_consensus.cc:729] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.413479 11151 consensus_queue.cc:260] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.414073 11151 ts_tablet_manager.cc:1398] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:27.415889 11152 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:27.416059 11152 ts_tablet_manager.cc:1367] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.001s
I20240503 08:14:27.417130 11152 raft_consensus.cc:348] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.417384 11152 raft_consensus.cc:374] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.417465 11152 raft_consensus.cc:729] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.417707 11152 consensus_queue.cc:260] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.418292 11152 ts_tablet_manager.cc:1398] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:27.420835 11153 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:27.420994 11153 ts_tablet_manager.cc:1367] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 08:14:27.421742 11153 raft_consensus.cc:348] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.422001 11153 raft_consensus.cc:374] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.422091 11153 raft_consensus.cc:729] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.422317 11153 consensus_queue.cc:260] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.422639 11166 raft_consensus.cc:482] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:27.422763 11166 raft_consensus.cc:504] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.423902 11166 leader_election.cc:288] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:27.424830 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:27.425032 9534 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 0.
I20240503 08:14:27.425655 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:27.425930 9798 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 0.
I20240503 08:14:27.427292 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:27.427524 9664 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 0.
I20240503 08:14:27.427510 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:27.427721 10060 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 0.
I20240503 08:14:27.429214 9864 leader_election.cc:302] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5; no voters:
I20240503 08:14:27.429809 11166 raft_consensus.cc:2781] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:27.429929 11166 raft_consensus.cc:482] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:27.430022 11166 raft_consensus.cc:3037] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.431668 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
W20240503 08:14:27.432850 9863 leader_election.cc:341] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:27.434832 11166 raft_consensus.cc:504] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.449487 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:27.449739 9534 raft_consensus.cc:3037] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.453298 9534 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 1.
I20240503 08:14:27.455081 11140 raft_consensus.cc:482] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:27.455267 11140 raft_consensus.cc:504] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.457394 11140 leader_election.cc:288] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:27.458117 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:27.459486 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:27.459635 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:27.459697 9664 raft_consensus.cc:3037] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.459856 10060 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:27.460551 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:27.460798 9798 raft_consensus.cc:3037] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.462705 9664 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 1.
I20240503 08:14:27.463047 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:27.463297 9665 raft_consensus.cc:2379] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate 06f1b17731834350a289de93abab8542 in this term.
I20240503 08:14:27.463901 9798 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 1.
I20240503 08:14:27.464708 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
W20240503 08:14:27.464743 10256 leader_election.cc:341] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:27.464977 9797 raft_consensus.cc:2379] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate 06f1b17731834350a289de93abab8542 in this term.
I20240503 08:14:27.465415 9864 leader_election.cc:302] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:27.465916 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:27.466288 9534 raft_consensus.cc:2379] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate 06f1b17731834350a289de93abab8542 in this term.
I20240503 08:14:27.466778 10255 leader_election.cc:302] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51; no voters: 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468
I20240503 08:14:27.467254 11140 raft_consensus.cc:3037] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.469202 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:27.469411 10060 raft_consensus.cc:3037] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.470118 11140 raft_consensus.cc:2726] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:27.470811 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:27.471043 10320 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 1.
I20240503 08:14:27.471910 9927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:27.472185 9927 raft_consensus.cc:2379] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate 06f1b17731834350a289de93abab8542 in this term.
I20240503 08:14:27.472398 10060 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 1.
I20240503 08:14:27.474249 11166 leader_election.cc:288] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [CANDIDATE]: Term 1 election: Requested vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:27.474560 11166 raft_consensus.cc:2781] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:27.474556 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:27.474745 11166 raft_consensus.cc:686] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 LEADER]: Becoming Leader. State: Replica: 06f1b17731834350a289de93abab8542, State: Running, Role: LEADER
W20240503 08:14:27.475239 9863 leader_election.cc:341] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:27.475376 11166 consensus_queue.cc:237] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.477577 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "06f1b17731834350a289de93abab8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:27.480508 10320 raft_consensus.cc:2454] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f1b17731834350a289de93abab8542 in term 1.
I20240503 08:14:27.486351 11155 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:27.486608 11155 ts_tablet_manager.cc:1367] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.108s user 0.000s sys 0.018s
I20240503 08:14:27.487422 11155 raft_consensus.cc:348] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.487821 11155 raft_consensus.cc:374] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.487936 11155 raft_consensus.cc:729] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.488250 11155 consensus_queue.cc:260] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.489110 11155 ts_tablet_manager.cc:1398] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:27.489477 11155 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:27.491055 11155 tablet_bootstrap.cc:654] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:27.496336 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:27.496562 9929 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 0.
I20240503 08:14:27.498088 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:27.499203 10060 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 0.
I20240503 08:14:27.502795 9391 catalog_manager.cc:5484] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06f1b17731834350a289de93abab8542 (127.24.123.4). New cstate: current_term: 1 leader_uuid: "06f1b17731834350a289de93abab8542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:27.511886 11153 ts_tablet_manager.cc:1398] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.091s user 0.017s sys 0.050s
I20240503 08:14:27.512768 11161 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:27.513033 11161 ts_tablet_manager.cc:1367] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.111s user 0.034s sys 0.059s
I20240503 08:14:27.513998 11155 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:27.513957 11161 raft_consensus.cc:348] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.514217 11155 ts_tablet_manager.cc:1367] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.013s
I20240503 08:14:27.514389 11161 raft_consensus.cc:374] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.514544 11161 raft_consensus.cc:729] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.514670 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:27.514866 10191 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 0.
I20240503 08:14:27.515378 9601 leader_election.cc:302] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:27.515390 11155 raft_consensus.cc:348] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.515622 11155 raft_consensus.cc:374] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:27.515749 11155 raft_consensus.cc:729] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:27.515872 11157 raft_consensus.cc:2781] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:27.515978 11157 raft_consensus.cc:482] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:27.516075 11157 raft_consensus.cc:3037] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.516003 11155 consensus_queue.cc:260] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.516568 11155 ts_tablet_manager.cc:1398] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:27.518796 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:27.519047 9798 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 0.
I20240503 08:14:27.518855 11157 raft_consensus.cc:504] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.520874 11190 raft_consensus.cc:482] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:27.521070 11190 raft_consensus.cc:504] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.521471 11140 raft_consensus.cc:482] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:27.521528 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:27.521731 10060 raft_consensus.cc:3037] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.521644 11140 raft_consensus.cc:504] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.522317 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:27.522300 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:27.522485 9929 raft_consensus.cc:3037] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.522516 9534 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 0.
I20240503 08:14:27.522681 11140 leader_election.cc:288] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:27.524156 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:27.524307 10060 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 1.
I20240503 08:14:27.524606 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:27.524792 10320 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 0.
I20240503 08:14:27.525025 9929 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 1.
I20240503 08:14:27.525053 9927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:27.525290 9927 raft_consensus.cc:2379] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in this term.
I20240503 08:14:27.525242 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:27.525460 9534 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:27.525771 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:27.526021 9664 raft_consensus.cc:2379] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cb18b8b10f944e39829faf16e061fd6 in current term 1: Already voted for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in this term.
I20240503 08:14:27.525965 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:27.526190 9798 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:27.526177 11157 leader_election.cc:288] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [CANDIDATE]: Term 1 election: Requested vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:27.524641 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:27.526543 10191 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:27.526687 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:27.527164 10256 leader_election.cc:302] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468; no voters: 06f1b17731834350a289de93abab8542, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1
I20240503 08:14:27.514901 11161 consensus_queue.cc:260] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.527719 11190 leader_election.cc:288] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:27.527757 11161 ts_tablet_manager.cc:1398] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 08:14:27.528152 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:27.528331 9534 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:27.528350 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:27.528507 10320 raft_consensus.cc:3037] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.528528 9927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:27.528692 9927 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:27.526835 10188 raft_consensus.cc:3037] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.529503 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:27.529697 9797 raft_consensus.cc:3037] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.531106 10320 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 1.
I20240503 08:14:27.531404 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:27.531600 9664 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:27.531864 10188 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 1.
I20240503 08:14:27.532104 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:27.532292 9797 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 1.
I20240503 08:14:27.532289 10060 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:27.532285 9600 leader_election.cc:302] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:27.532418 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8506211351284b7a8e2a166b8083977a" candidate_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:27.532636 9534 raft_consensus.cc:3037] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.532781 10126 leader_election.cc:302] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:27.533049 11157 raft_consensus.cc:2781] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:27.533138 11190 raft_consensus.cc:2781] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:27.533231 11190 raft_consensus.cc:482] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:27.533298 11190 raft_consensus.cc:3037] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.533710 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:27.533906 9798 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:27.534250 11140 raft_consensus.cc:2741] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 08:14:27.534539 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:27.534771 10320 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:27.535401 9534 raft_consensus.cc:2454] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1eb5e69c6264a20a6e8297cc5deb8d1 in term 1.
I20240503 08:14:27.535987 11157 raft_consensus.cc:686] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 LEADER]: Becoming Leader. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Running, Role: LEADER
I20240503 08:14:27.536288 11190 raft_consensus.cc:504] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.536435 11157 consensus_queue.cc:237] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:27.537374 11190 leader_election.cc:288] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 election: Requested vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:27.537568 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:27.537590 9927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:27.537746 9927 raft_consensus.cc:3037] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.537778 10320 raft_consensus.cc:3037] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.537750 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:27.537799 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:27.537927 10060 raft_consensus.cc:3037] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.537950 9798 raft_consensus.cc:3037] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.540367 10320 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:27.540449 9927 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:27.540535 9798 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:27.540535 10060 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:27.541002 10124 leader_election.cc:302] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:27.541150 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:27.541322 9534 raft_consensus.cc:3037] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.541347 11190 raft_consensus.cc:2781] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:27.541378 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f0afdc01d4d03917b9040c2053426" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:27.541563 9664 raft_consensus.cc:3037] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.541718 11190 raft_consensus.cc:686] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Becoming Leader. State: Replica: fa60e870de844325bad4b9185eaee468, State: Running, Role: LEADER
I20240503 08:14:27.542027 11190 consensus_queue.cc:237] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.543830 9534 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:27.544415 9664 raft_consensus.cc:2454] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:27.546422 9390 catalog_manager.cc:5484] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa60e870de844325bad4b9185eaee468 (127.24.123.6). New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:27.547525 9391 catalog_manager.cc:5484] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2). New cstate: current_term: 1 leader_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:27.752394 11190 raft_consensus.cc:482] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:27.752666 11190 raft_consensus.cc:504] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:27.753894 11190 leader_election.cc:288] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:27.754338 9927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:27.754338 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:27.754616 9534 raft_consensus.cc:2379] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate 06f1b17731834350a289de93abab8542 in this term.
I20240503 08:14:27.754557 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:27.754844 9798 raft_consensus.cc:2379] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate 06f1b17731834350a289de93abab8542 in this term.
I20240503 08:14:27.755493 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:27.755532 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:27.755764 9664 raft_consensus.cc:2379] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate 06f1b17731834350a289de93abab8542 in this term.
I20240503 08:14:27.755774 10320 raft_consensus.cc:2379] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate 06f1b17731834350a289de93abab8542 in this term.
I20240503 08:14:27.755770 10060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd41cbc6c69441c1af5c36c382cfe366" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:27.756040 10060 raft_consensus.cc:2379] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate 06f1b17731834350a289de93abab8542 in this term.
I20240503 08:14:27.756253 10126 leader_election.cc:302] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: fa60e870de844325bad4b9185eaee468; no voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1
I20240503 08:14:27.756695 11190 raft_consensus.cc:3037] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:27.759090 11190 raft_consensus.cc:2726] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:27.869530 11186 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:27.905004 11186 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:27.933640 11186 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:27.942595 11166 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:27.950070 11195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:27.950574 11195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:14:27.956477 11166 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:27.961665 11195 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:27.967908 11204 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:27.977852 11157 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.002826 11190 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.004487 10912 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 08:14:28.010824 10912 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.015467 10912 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.022048 10912 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.022735 11157 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.025049 11190 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.044335 11157 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.049279 11157 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.057296 11242 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:28.064967 11240 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 429.428 ms
time per row: 0.214714 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:14:29.469359 9389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55074:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:29.470057 9389 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_3 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:14:29.479473 9907 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe97b80e1294c4791560daf991bfcdb (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.479972 9907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.479871 10169 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe97b80e1294c4791560daf991bfcdb (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.480042 10300 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe97b80e1294c4791560daf991bfcdb (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.480294 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.480438 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.480842 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe97b80e1294c4791560daf991bfcdb (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.481228 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.483578 10040 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe97b80e1294c4791560daf991bfcdb (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.483986 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.487756 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe97b80e1294c4791560daf991bfcdb (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.488162 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.488752 9776 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe97b80e1294c4791560daf991bfcdb (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.489184 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.491729 9909 tablet_service.cc:1467] Processing CreateTablet for tablet c9306edc6261425cb2c7bef9caf9135b (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.492116 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9306edc6261425cb2c7bef9caf9135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.494206 10038 tablet_service.cc:1467] Processing CreateTablet for tablet c9306edc6261425cb2c7bef9caf9135b (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.494473 10170 tablet_service.cc:1467] Processing CreateTablet for tablet c9306edc6261425cb2c7bef9caf9135b (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.494580 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9306edc6261425cb2c7bef9caf9135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.494846 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9306edc6261425cb2c7bef9caf9135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.504565 10301 tablet_service.cc:1467] Processing CreateTablet for tablet c9306edc6261425cb2c7bef9caf9135b (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.505014 10301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9306edc6261425cb2c7bef9caf9135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.506410 10171 tablet_service.cc:1467] Processing CreateTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.506776 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.506865 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.507340 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.507977 9647 tablet_service.cc:1467] Processing CreateTablet for tablet c9306edc6261425cb2c7bef9caf9135b (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.508373 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9306edc6261425cb2c7bef9caf9135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.509775 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.510150 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.511250 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.511607 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.511529 9514 tablet_service.cc:1467] Processing CreateTablet for tablet c9306edc6261425cb2c7bef9caf9135b (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.511888 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9306edc6261425cb2c7bef9caf9135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.513625 9777 tablet_service.cc:1467] Processing CreateTablet for tablet c9306edc6261425cb2c7bef9caf9135b (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.514051 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9306edc6261425cb2c7bef9caf9135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.515101 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.515450 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.518330 11315 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:29.522118 11317 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:29.522215 11314 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:29.523696 11314 tablet_bootstrap.cc:654] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.523732 11317 tablet_bootstrap.cc:654] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.524770 10039 tablet_service.cc:1467] Processing CreateTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.524966 11315 tablet_bootstrap.cc:654] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.525171 10039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.526131 10300 tablet_service.cc:1467] Processing CreateTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:29.526587 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:29.534796 11319 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:29.536139 11314 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:29.536178 11315 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:29.536325 11315 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 08:14:29.536834 11319 tablet_bootstrap.cc:654] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.537066 11314 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240503 08:14:29.537307 11318 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:29.538707 11318 tablet_bootstrap.cc:654] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.538512 11314 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.538512 11315 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.538946 11314 raft_consensus.cc:374] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.539083 11314 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.539377 11314 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.540249 11314 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:29.540922 11313 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:29.541254 11314 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:29.538797 11315 raft_consensus.cc:374] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.541883 11315 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.542428 11315 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.544014 11313 tablet_bootstrap.cc:654] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.544472 11314 tablet_bootstrap.cc:654] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.544538 11318 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:29.544744 11318 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:14:29.545495 11318 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.545796 11318 raft_consensus.cc:374] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.545959 11318 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.546269 11318 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.549197 11318 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:29.549221 11315 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:14:29.550639 11318 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:29.552381 11318 tablet_bootstrap.cc:654] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.555769 11315 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:29.556001 11314 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:29.556228 11314 ts_tablet_manager.cc:1367] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20240503 08:14:29.557241 11314 raft_consensus.cc:348] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.557595 11314 raft_consensus.cc:374] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.557710 11314 raft_consensus.cc:729] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.558035 11314 consensus_queue.cc:260] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.558774 11314 ts_tablet_manager.cc:1398] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:29.559446 11314 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:29.560513 11313 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:29.560710 11313 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.003s
I20240503 08:14:29.561489 11313 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.561887 11313 raft_consensus.cc:374] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.562023 11313 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.562180 11316 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:29.562335 11313 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.563161 11313 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:29.563504 11313 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:29.565007 11314 tablet_bootstrap.cc:654] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.565073 11317 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:29.565300 11317 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.003s
I20240503 08:14:29.566596 11313 tablet_bootstrap.cc:654] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.566987 11315 tablet_bootstrap.cc:654] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.567626 11317 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.568202 11319 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:29.568395 11319 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.010s
I20240503 08:14:29.569008 11316 tablet_bootstrap.cc:654] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.569263 11317 raft_consensus.cc:374] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.569170 11319 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.569427 11317 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.569551 11319 raft_consensus.cc:374] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.569685 11319 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.569706 11317 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.570004 11319 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.570343 11317 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:14:29.571389 11319 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:14:29.571749 11319 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:29.573372 11319 tablet_bootstrap.cc:654] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.576205 11314 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:29.576402 11314 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240503 08:14:29.577199 11314 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.577534 11314 raft_consensus.cc:374] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.577677 11314 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.578133 11316 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:29.578002 11314 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.578346 11316 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240503 08:14:29.578722 11314 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:29.578979 11316 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.579308 11316 raft_consensus.cc:374] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.579443 11316 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.579777 11313 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:29.579798 11315 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:29.579972 11315 ts_tablet_manager.cc:1367] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.001s
I20240503 08:14:29.579761 11316 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.579995 11313 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20240503 08:14:29.580634 11315 raft_consensus.cc:348] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.580906 11315 raft_consensus.cc:374] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.580989 11315 raft_consensus.cc:729] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.580799 11313 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.581337 11313 raft_consensus.cc:374] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.581229 11315 consensus_queue.cc:260] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.581460 11313 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.581804 11315 ts_tablet_manager.cc:1398] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:29.581842 11313 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.582119 11315 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:29.582631 11313 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:29.582990 11313 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:29.583627 11315 tablet_bootstrap.cc:654] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.584468 11313 tablet_bootstrap.cc:654] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.584652 11317 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:29.584800 11316 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 08:14:29.586146 11317 tablet_bootstrap.cc:654] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.587406 11318 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:29.587615 11316 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:29.587601 11318 ts_tablet_manager.cc:1367] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20240503 08:14:29.588327 11318 raft_consensus.cc:348] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.588606 11318 raft_consensus.cc:374] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.588682 11318 raft_consensus.cc:729] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.588943 11318 consensus_queue.cc:260] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.589473 11318 ts_tablet_manager.cc:1398] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:29.589658 11316 tablet_bootstrap.cc:654] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.589777 11318 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:29.592381 11318 tablet_bootstrap.cc:654] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.594265 11313 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:29.594455 11313 ts_tablet_manager.cc:1367] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240503 08:14:29.595206 11313 raft_consensus.cc:348] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.595400 11313 raft_consensus.cc:374] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.595479 11313 raft_consensus.cc:729] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.595734 11313 consensus_queue.cc:260] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.596498 11313 ts_tablet_manager.cc:1398] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:29.596634 11318 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:29.596850 11318 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:14:29.597482 11318 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.597666 11318 raft_consensus.cc:374] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.597749 11318 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.597981 11318 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.598562 11318 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:29.599361 11315 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:29.599371 11319 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:29.599540 11319 ts_tablet_manager.cc:1367] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.002s
I20240503 08:14:29.599567 11315 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20240503 08:14:29.600255 11315 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.600239 11319 raft_consensus.cc:348] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.600493 11315 raft_consensus.cc:374] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.600509 11319 raft_consensus.cc:374] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.600571 11319 raft_consensus.cc:729] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.600574 11315 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.600807 11315 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.600807 11319 consensus_queue.cc:260] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.601337 11319 ts_tablet_manager.cc:1398] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:14:29.601665 11315 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:29.602846 11319 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:29.604259 11319 tablet_bootstrap.cc:654] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.605957 11319 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:29.606091 11319 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:14:29.606712 11319 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.606932 11319 raft_consensus.cc:374] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.607028 11319 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.607231 11319 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.607714 11319 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:14:29.615296 11316 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:29.615300 11317 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:29.615464 11316 ts_tablet_manager.cc:1367] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.005s
I20240503 08:14:29.615463 11317 ts_tablet_manager.cc:1367] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.003s
I20240503 08:14:29.616031 11316 raft_consensus.cc:348] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.616056 11317 raft_consensus.cc:348] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.616241 11317 raft_consensus.cc:374] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.616240 11316 raft_consensus.cc:374] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.616310 11317 raft_consensus.cc:729] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.616312 11316 raft_consensus.cc:729] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.616551 11317 consensus_queue.cc:260] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.616547 11316 consensus_queue.cc:260] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.617029 11317 ts_tablet_manager.cc:1398] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:14:29.617165 11316 ts_tablet_manager.cc:1398] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:29.617486 11317 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:29.617504 11316 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:29.618782 11316 tablet_bootstrap.cc:654] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.620446 11317 tablet_bootstrap.cc:654] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:29.621860 10989 raft_consensus.cc:482] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:29.622004 10989 raft_consensus.cc:504] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.622340 11317 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:29.622469 11317 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:14:29.623170 10989 leader_election.cc:288] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:29.623178 11317 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.623275 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:29.623391 11317 raft_consensus.cc:374] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.623474 11317 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.623528 9928 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:29.623822 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:29.623752 11317 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.624030 10058 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:29.624305 11317 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:29.624212 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:29.624442 9664 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:29.624676 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:29.624884 10188 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:29.625345 11265 raft_consensus.cc:482] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:29.625499 11265 raft_consensus.cc:504] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.626006 9469 leader_election.cc:302] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:29.626710 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:29.626780 11265 leader_election.cc:288] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:29.626912 10320 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:29.627388 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:29.627557 10320 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:29.627605 10989 raft_consensus.cc:2781] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:29.627697 10989 raft_consensus.cc:482] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:29.627765 10989 raft_consensus.cc:3037] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.627902 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:29.627971 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:29.628142 9797 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:29.628146 9796 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:29.628839 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:29.629019 10058 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:29.629351 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:29.629554 9664 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:29.629921 10126 leader_election.cc:302] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:29.630174 11265 raft_consensus.cc:2781] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:29.630287 11265 raft_consensus.cc:482] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:29.630376 11265 raft_consensus.cc:3037] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.630436 10989 raft_consensus.cc:504] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.630897 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:29.631143 9928 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:29.631670 10989 leader_election.cc:288] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Requested vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:29.632550 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:29.632737 10320 raft_consensus.cc:3037] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.632745 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:29.632921 9533 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:29.633036 11265 raft_consensus.cc:504] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.633479 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:29.633652 9796 raft_consensus.cc:3037] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.634120 11265 leader_election.cc:288] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 election: Requested vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:29.635452 10320 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:29.635582 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:29.635782 9664 raft_consensus.cc:3037] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.636215 9796 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:29.637496 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:29.637691 9665 raft_consensus.cc:3037] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.638326 9664 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:29.639420 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:29.639662 10059 raft_consensus.cc:3037] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.639793 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:29.639973 10058 raft_consensus.cc:3037] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.640110 9665 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:29.641022 11316 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:29.641173 11316 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20240503 08:14:29.641281 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:29.641489 9796 raft_consensus.cc:3037] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.641918 11316 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.642165 11316 raft_consensus.cc:374] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:29.642251 11316 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:29.642400 10059 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:29.642479 11316 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.642705 10058 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:29.643069 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:29.643193 11316 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:29.643311 9928 raft_consensus.cc:3037] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.644281 9796 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:29.644951 10124 leader_election.cc:302] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:29.645313 11265 raft_consensus.cc:2781] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:29.645465 11265 raft_consensus.cc:686] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Becoming Leader. State: Replica: fa60e870de844325bad4b9185eaee468, State: Running, Role: LEADER
I20240503 08:14:29.645884 9928 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:29.646332 9468 leader_election.cc:302] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:29.645823 11265 consensus_queue.cc:237] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.646939 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:29.647074 10989 raft_consensus.cc:2781] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:29.647137 9533 raft_consensus.cc:3037] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.648663 10989 raft_consensus.cc:686] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Becoming Leader. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Running, Role: LEADER
I20240503 08:14:29.649065 10989 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:29.649600 9533 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:29.651103 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:29.651314 10320 raft_consensus.cc:3037] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.653255 10320 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:29.655339 9389 catalog_manager.cc:5484] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa60e870de844325bad4b9185eaee468 (127.24.123.6). New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:29.662137 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9306edc6261425cb2c7bef9caf9135b" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:29.662355 9928 raft_consensus.cc:3037] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.662393 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe97b80e1294c4791560daf991bfcdb" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:29.662611 10188 raft_consensus.cc:3037] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.663326 9389 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:29.664577 9928 raft_consensus.cc:2454] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:29.665211 10188 raft_consensus.cc:2454] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:29.772370 11307 raft_consensus.cc:482] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:29.772605 11307 raft_consensus.cc:504] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.773710 11307 leader_election.cc:288] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:29.773943 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:29.774086 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:29.774191 9665 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:29.774163 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:29.774359 9533 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:29.774405 9928 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:29.774395 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:29.774628 10058 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:29.775133 9733 leader_election.cc:302] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:29.775254 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:29.775372 11307 raft_consensus.cc:2781] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:29.775483 11307 raft_consensus.cc:482] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:29.775466 10188 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:29.775571 11307 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.775990 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:29.776192 10320 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:29.777868 11307 raft_consensus.cc:504] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.779047 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:29.779110 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:29.779247 9665 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.779234 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:29.779222 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:29.779295 9928 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.779398 10188 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.779417 9533 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.781922 9928 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:29.781934 9665 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:29.782023 10188 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:29.782117 9533 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:29.782469 9731 leader_election.cc:302] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:29.782788 11307 leader_election.cc:288] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Requested vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:29.782792 11347 raft_consensus.cc:2781] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:29.783129 11347 raft_consensus.cc:686] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Becoming Leader. State: Replica: 8de08e679a904ee48ba659861c730725, State: Running, Role: LEADER
I20240503 08:14:29.783244 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:29.783452 10320 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.783394 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:29.783597 10058 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:29.783453 11347 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:29.786120 10320 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:29.786342 10058 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:29.788682 9389 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de08e679a904ee48ba659861c730725 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:30.040647 11342 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.074841 11265 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.076115 11265 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.111394 11342 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.111878 11342 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.112253 11342 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.155249 10989 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.165601 11347 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:30.168051 11349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:30.168391 11349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:14:30.175134 11347 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:30.176726 11349 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:30.199448 10989 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.207319 11347 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 08:14:30.213268 11307 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.214202 11307 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.216888 11307 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.234968 10989 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.238381 10989 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:30.238863 10989 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240503 08:14:30.275643 11396 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 446.606 ms
time per row: 0.223303 ms
I20240503 08:14:30.862388 9389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55074:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:30.863265 9389 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_4 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:14:30.872220 9515 tablet_service.cc:1467] Processing CreateTablet for tablet a61f59d188824bf3b610771b2fc05f06 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.872376 9646 tablet_service.cc:1467] Processing CreateTablet for tablet a61f59d188824bf3b610771b2fc05f06 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.872700 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f59d188824bf3b610771b2fc05f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.872839 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f59d188824bf3b610771b2fc05f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.873294 10169 tablet_service.cc:1467] Processing CreateTablet for tablet a61f59d188824bf3b610771b2fc05f06 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.873757 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f59d188824bf3b610771b2fc05f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.875300 10301 tablet_service.cc:1467] Processing CreateTablet for tablet f13c7cfe35b3403f87940a124e6d9cda (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.875689 10301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c7cfe35b3403f87940a124e6d9cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.877669 10302 tablet_service.cc:1467] Processing CreateTablet for tablet fdb88bebfdab4e54abc5dba5834a3298 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.878034 10302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb88bebfdab4e54abc5dba5834a3298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.879287 9778 tablet_service.cc:1467] Processing CreateTablet for tablet a61f59d188824bf3b610771b2fc05f06 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.879734 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f59d188824bf3b610771b2fc05f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.880228 9777 tablet_service.cc:1467] Processing CreateTablet for tablet fdb88bebfdab4e54abc5dba5834a3298 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.880622 9776 tablet_service.cc:1467] Processing CreateTablet for tablet f13c7cfe35b3403f87940a124e6d9cda (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.881677 10170 tablet_service.cc:1467] Processing CreateTablet for tablet f13c7cfe35b3403f87940a124e6d9cda (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.882045 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c7cfe35b3403f87940a124e6d9cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.882364 9647 tablet_service.cc:1467] Processing CreateTablet for tablet f13c7cfe35b3403f87940a124e6d9cda (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.882750 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c7cfe35b3403f87940a124e6d9cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.883247 10300 tablet_service.cc:1467] Processing CreateTablet for tablet a61f59d188824bf3b610771b2fc05f06 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.883621 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f59d188824bf3b610771b2fc05f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.887265 9645 tablet_service.cc:1467] Processing CreateTablet for tablet fdb88bebfdab4e54abc5dba5834a3298 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.887641 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb88bebfdab4e54abc5dba5834a3298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.887807 9909 tablet_service.cc:1467] Processing CreateTablet for tablet a61f59d188824bf3b610771b2fc05f06 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.888263 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f59d188824bf3b610771b2fc05f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.888571 9513 tablet_service.cc:1467] Processing CreateTablet for tablet f13c7cfe35b3403f87940a124e6d9cda (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.889042 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c7cfe35b3403f87940a124e6d9cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.891259 9907 tablet_service.cc:1467] Processing CreateTablet for tablet f13c7cfe35b3403f87940a124e6d9cda (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.891288 9514 tablet_service.cc:1467] Processing CreateTablet for tablet fdb88bebfdab4e54abc5dba5834a3298 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.891628 9907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c7cfe35b3403f87940a124e6d9cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.891667 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb88bebfdab4e54abc5dba5834a3298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.893330 10040 tablet_service.cc:1467] Processing CreateTablet for tablet f13c7cfe35b3403f87940a124e6d9cda (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.893744 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c7cfe35b3403f87940a124e6d9cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.895248 9908 tablet_service.cc:1467] Processing CreateTablet for tablet fdb88bebfdab4e54abc5dba5834a3298 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.895287 10038 tablet_service.cc:1467] Processing CreateTablet for tablet a61f59d188824bf3b610771b2fc05f06 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.895613 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb88bebfdab4e54abc5dba5834a3298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.895732 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f59d188824bf3b610771b2fc05f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.897547 10039 tablet_service.cc:1467] Processing CreateTablet for tablet fdb88bebfdab4e54abc5dba5834a3298 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.897935 10039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb88bebfdab4e54abc5dba5834a3298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.899164 10171 tablet_service.cc:1467] Processing CreateTablet for tablet fdb88bebfdab4e54abc5dba5834a3298 (DEFAULT_TABLE table=rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:30.899595 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb88bebfdab4e54abc5dba5834a3298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.880623 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb88bebfdab4e54abc5dba5834a3298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.912371 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c7cfe35b3403f87940a124e6d9cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:30.915210 11452 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:30.917114 11452 tablet_bootstrap.cc:654] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.942358 11454 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:30.945796 11449 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:30.946007 11454 tablet_bootstrap.cc:654] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.947408 11449 tablet_bootstrap.cc:654] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.949343 11453 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:30.949493 11451 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:30.951365 11453 tablet_bootstrap.cc:654] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.951444 11451 tablet_bootstrap.cc:654] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.952821 11450 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:30.953820 11449 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:30.953979 11449 ts_tablet_manager.cc:1367] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:14:30.954762 11449 raft_consensus.cc:348] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.955205 11449 raft_consensus.cc:374] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.955345 11449 raft_consensus.cc:729] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.955660 11451 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:30.955688 11452 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:30.955837 11452 ts_tablet_manager.cc:1367] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.012s
I20240503 08:14:30.955847 11451 ts_tablet_manager.cc:1367] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:14:30.955654 11449 consensus_queue.cc:260] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.956545 11451 raft_consensus.cc:348] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.956856 11451 raft_consensus.cc:374] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.956897 11450 tablet_bootstrap.cc:654] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.956985 11451 raft_consensus.cc:729] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.956868 11452 raft_consensus.cc:348] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.957228 11451 consensus_queue.cc:260] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.957451 11452 raft_consensus.cc:374] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.957839 11451 ts_tablet_manager.cc:1398] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:30.957897 11453 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:30.958087 11453 ts_tablet_manager.cc:1367] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:14:30.958240 11451 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:30.958809 11454 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:30.959239 11449 ts_tablet_manager.cc:1398] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:14:30.959784 11454 ts_tablet_manager.cc:1367] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 08:14:30.961282 11451 tablet_bootstrap.cc:654] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.961292 11454 raft_consensus.cc:348] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.961619 11454 raft_consensus.cc:374] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.961841 11449 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:30.961814 11454 raft_consensus.cc:729] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.962038 11454 consensus_queue.cc:260] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.962589 11453 raft_consensus.cc:348] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.963472 11454 ts_tablet_manager.cc:1398] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:30.963707 11449 tablet_bootstrap.cc:654] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.963845 11454 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:30.964974 11453 raft_consensus.cc:374] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.965163 11453 raft_consensus.cc:729] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.966287 11454 tablet_bootstrap.cc:654] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.967490 11453 consensus_queue.cc:260] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.967999 11452 raft_consensus.cc:729] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.968325 11452 consensus_queue.cc:260] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.968760 11454 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:30.968928 11454 ts_tablet_manager.cc:1367] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:14:30.969601 11454 raft_consensus.cc:348] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.969864 11454 raft_consensus.cc:374] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.969941 11454 raft_consensus.cc:729] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.970755 11451 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:30.971002 11451 ts_tablet_manager.cc:1367] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240503 08:14:30.970170 11454 consensus_queue.cc:260] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.971959 11454 ts_tablet_manager.cc:1398] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:30.972559 11452 ts_tablet_manager.cc:1398] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240503 08:14:30.972584 11451 raft_consensus.cc:348] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.972932 11454 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:30.972972 11451 raft_consensus.cc:374] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.973124 11451 raft_consensus.cc:729] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.973412 11451 consensus_queue.cc:260] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.974242 11451 ts_tablet_manager.cc:1398] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:30.974596 11451 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:30.974769 11452 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:30.974793 11454 tablet_bootstrap.cc:654] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.976361 11452 tablet_bootstrap.cc:654] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.976924 11451 tablet_bootstrap.cc:654] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.978199 11449 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:30.978399 11449 ts_tablet_manager.cc:1367] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20240503 08:14:30.979174 11449 raft_consensus.cc:348] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.979460 11449 raft_consensus.cc:374] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.979543 11449 raft_consensus.cc:729] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.979813 11449 consensus_queue.cc:260] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.981905 11449 ts_tablet_manager.cc:1398] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:30.982259 11449 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:30.983280 11452 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:30.983458 11452 ts_tablet_manager.cc:1367] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:14:30.983597 11453 ts_tablet_manager.cc:1398] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240503 08:14:30.985042 11453 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:30.986344 11449 tablet_bootstrap.cc:654] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.986511 11448 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:30.987577 11451 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:30.987660 11453 tablet_bootstrap.cc:654] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.987823 11451 ts_tablet_manager.cc:1367] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240503 08:14:30.988231 11454 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:30.988164 11452 raft_consensus.cc:348] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.988381 11454 ts_tablet_manager.cc:1367] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 08:14:30.988471 11452 raft_consensus.cc:374] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.988559 11452 raft_consensus.cc:729] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.988831 11452 consensus_queue.cc:260] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.989091 11454 raft_consensus.cc:348] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.989342 11454 raft_consensus.cc:374] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.989436 11454 raft_consensus.cc:729] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.989231 11451 raft_consensus.cc:348] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.989683 11451 raft_consensus.cc:374] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.989812 11451 raft_consensus.cc:729] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.989693 11454 consensus_queue.cc:260] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.990067 11452 ts_tablet_manager.cc:1398] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:14:30.990331 11452 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:30.990404 11454 ts_tablet_manager.cc:1398] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:30.990857 11451 consensus_queue.cc:260] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:30.991705 11451 ts_tablet_manager.cc:1398] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:14:30.992630 11449 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:30.992826 11449 ts_tablet_manager.cc:1367] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240503 08:14:30.993654 11452 tablet_bootstrap.cc:654] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.993997 11449 raft_consensus.cc:348] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.994570 11448 tablet_bootstrap.cc:654] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:30.995319 11449 raft_consensus.cc:374] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.995489 11449 raft_consensus.cc:729] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.995781 11449 consensus_queue.cc:260] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.998312 11453 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:30.998477 11453 ts_tablet_manager.cc:1367] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240503 08:14:30.998988 11449 ts_tablet_manager.cc:1398] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:14:30.999089 11453 raft_consensus.cc:348] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:30.999317 11453 raft_consensus.cc:374] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:30.999392 11453 raft_consensus.cc:729] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:30.999624 11453 consensus_queue.cc:260] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.000873 11448 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:31.001081 11448 ts_tablet_manager.cc:1367] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240503 08:14:31.001852 11448 raft_consensus.cc:348] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:31.002252 11448 raft_consensus.cc:374] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:31.002388 11448 raft_consensus.cc:729] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:31.002836 11450 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:31.003259 11450 ts_tablet_manager.cc:1367] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20240503 08:14:31.002653 11448 consensus_queue.cc:260] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:31.004026 11448 ts_tablet_manager.cc:1398] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:14:31.004390 11448 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:31.004070 11450 raft_consensus.cc:348] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:31.005097 11450 raft_consensus.cc:374] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:31.005203 11450 raft_consensus.cc:729] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:31.005436 11450 consensus_queue.cc:260] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:31.006140 11450 ts_tablet_manager.cc:1398] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:31.006330 11448 tablet_bootstrap.cc:654] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:31.006455 11450 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:31.008100 11450 tablet_bootstrap.cc:654] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:31.015156 11453 ts_tablet_manager.cc:1398] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:14:31.017894 11453 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:31.018388 11450 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:31.018573 11450 ts_tablet_manager.cc:1367] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240503 08:14:31.019600 11453 tablet_bootstrap.cc:654] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:31.019407 11450 raft_consensus.cc:348] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.019698 11450 raft_consensus.cc:374] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:31.019776 11450 raft_consensus.cc:729] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:31.020107 11450 consensus_queue.cc:260] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.020913 11450 ts_tablet_manager.cc:1398] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:14:31.021306 11450 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:31.023444 11450 tablet_bootstrap.cc:654] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:31.023551 11452 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:31.023707 11452 ts_tablet_manager.cc:1367] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.007s
I20240503 08:14:31.024533 11452 raft_consensus.cc:348] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.026608 11452 raft_consensus.cc:374] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:31.026717 11452 raft_consensus.cc:729] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:31.027077 11452 consensus_queue.cc:260] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.027349 11453 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:31.027503 11453 ts_tablet_manager.cc:1367] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240503 08:14:31.028239 11342 raft_consensus.cc:482] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:31.028412 11342 raft_consensus.cc:504] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:31.029680 11342 leader_election.cc:288] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:31.028265 11453 raft_consensus.cc:348] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.029814 11453 raft_consensus.cc:374] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:31.029938 11453 raft_consensus.cc:729] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:31.030196 11453 consensus_queue.cc:260] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.030776 11453 ts_tablet_manager.cc:1398] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:31.031093 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:31.031308 9535 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:31.031837 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:31.032052 9797 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:31.032500 11448 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:31.032462 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:31.032728 10321 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:31.033059 11448 ts_tablet_manager.cc:1367] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.021s
I20240503 08:14:31.033304 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:31.033485 9928 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:31.033902 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:31.034152 9664 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:31.034188 11448 raft_consensus.cc:348] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.034467 11448 raft_consensus.cc:374] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:31.034534 11448 raft_consensus.cc:729] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:31.034518 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:31.034730 10058 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:31.035056 11448 consensus_queue.cc:260] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.035485 10127 leader_election.cc:302] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:31.035648 11448 ts_tablet_manager.cc:1398] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:31.036011 11448 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:31.036350 11342 raft_consensus.cc:2781] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:31.036468 11342 raft_consensus.cc:482] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:31.036549 11342 raft_consensus.cc:3037] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.037509 11448 tablet_bootstrap.cc:654] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:31.039424 11342 raft_consensus.cc:504] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:31.040748 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:31.040983 9535 raft_consensus.cc:3037] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.041821 11448 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:31.041950 11448 ts_tablet_manager.cc:1367] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 08:14:31.042486 11448 raft_consensus.cc:348] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.042667 11448 raft_consensus.cc:374] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:31.042788 11448 raft_consensus.cc:729] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:31.043037 11448 consensus_queue.cc:260] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.043653 11448 ts_tablet_manager.cc:1398] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:31.043835 9535 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:31.044955 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:31.045166 9664 raft_consensus.cc:3037] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.048032 9664 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:31.049057 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:31.049190 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:31.049276 9797 raft_consensus.cc:3037] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.049386 9928 raft_consensus.cc:3037] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.049876 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:31.050091 10058 raft_consensus.cc:3037] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.052093 9797 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:31.052275 9928 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:31.052592 10124 leader_election.cc:302] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:31.052659 10058 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:31.052938 11265 raft_consensus.cc:2781] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:31.053103 11265 raft_consensus.cc:686] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Becoming Leader. State: Replica: fa60e870de844325bad4b9185eaee468, State: Running, Role: LEADER
I20240503 08:14:31.053429 11265 consensus_queue.cc:237] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:31.056715 11342 leader_election.cc:288] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 election: Requested vote from peers b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:31.057287 11452 ts_tablet_manager.cc:1398] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240503 08:14:31.057286 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61f59d188824bf3b610771b2fc05f06" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:31.057488 10321 raft_consensus.cc:3037] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.059718 10321 raft_consensus.cc:2454] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 1.
I20240503 08:14:31.060421 11450 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:31.060595 11450 ts_tablet_manager.cc:1367] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.020s
I20240503 08:14:31.061398 11450 raft_consensus.cc:348] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.061728 11450 raft_consensus.cc:374] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:31.061812 11450 raft_consensus.cc:729] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:31.062062 11450 consensus_queue.cc:260] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.062914 11450 ts_tablet_manager.cc:1398] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:31.068059 11347 raft_consensus.cc:482] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:31.068246 11347 raft_consensus.cc:504] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.069469 11347 leader_election.cc:288] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:31.069648 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:31.069885 10058 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:31.070394 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:31.070607 9928 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:31.070668 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:31.070915 9535 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:31.071198 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:31.071259 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:31.071413 10191 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:31.071405 9731 leader_election.cc:302] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5; no voters:
I20240503 08:14:31.071470 9664 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:31.071669 11347 raft_consensus.cc:2781] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:31.071774 11347 raft_consensus.cc:482] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:31.071889 11347 raft_consensus.cc:3037] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.072798 9389 catalog_manager.cc:5484] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa60e870de844325bad4b9185eaee468 (127.24.123.6). New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:31.073557 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:31.073895 10321 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 0.
I20240503 08:14:31.074249 11347 raft_consensus.cc:504] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.075317 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:31.075340 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:31.075522 9928 raft_consensus.cc:3037] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.075523 10191 raft_consensus.cc:3037] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.078119 10191 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:31.078133 9928 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:31.079044 11347 leader_election.cc:288] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Requested vote from peers fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:31.079849 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:31.079953 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:31.080057 10321 raft_consensus.cc:3037] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.080157 9535 raft_consensus.cc:3037] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.080488 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:31.080667 9664 raft_consensus.cc:3037] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.082700 9535 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:31.082856 10321 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:31.083137 9731 leader_election.cc:302] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:31.083421 11347 raft_consensus.cc:2781] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:31.083413 9664 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:31.083564 11347 raft_consensus.cc:686] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Becoming Leader. State: Replica: 8de08e679a904ee48ba659861c730725, State: Running, Role: LEADER
I20240503 08:14:31.083896 11347 consensus_queue.cc:237] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.084059 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c7cfe35b3403f87940a124e6d9cda" candidate_uuid: "8de08e679a904ee48ba659861c730725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:31.084266 10058 raft_consensus.cc:3037] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.086782 10058 raft_consensus.cc:2454] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de08e679a904ee48ba659861c730725 in term 1.
I20240503 08:14:31.087833 9389 catalog_manager.cc:5484] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de08e679a904ee48ba659861c730725 (127.24.123.3). New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:31.144678 11327 raft_consensus.cc:482] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:31.144877 11327 raft_consensus.cc:504] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.146310 11327 leader_election.cc:288] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:31.146574 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:31.146827 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:31.146862 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:31.147001 10058 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:31.147085 10191 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:31.147154 9664 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:31.147521 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:31.147742 10257 leader_election.cc:302] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:31.147781 9535 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:31.148067 11327 raft_consensus.cc:2781] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:31.148182 11327 raft_consensus.cc:482] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:31.148154 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:31.148263 11327 raft_consensus.cc:3037] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.148272 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:31.148341 9928 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:31.148505 9797 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 0.
I20240503 08:14:31.150585 11327 raft_consensus.cc:504] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.151641 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:31.151731 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:31.151768 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:31.151917 10058 raft_consensus.cc:3037] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.151976 9928 raft_consensus.cc:3037] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.152016 10191 raft_consensus.cc:3037] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.152354 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:31.152621 9664 raft_consensus.cc:3037] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.154577 10058 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:31.154587 9928 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:31.155016 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:31.155220 9535 raft_consensus.cc:3037] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.155349 11327 leader_election.cc:288] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 election: Requested vote from peers fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:31.155349 9664 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:31.155386 10191 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:31.155687 10257 leader_election.cc:302] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:31.156015 11327 raft_consensus.cc:2781] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:31.155927 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb88bebfdab4e54abc5dba5834a3298" candidate_uuid: "0cb18b8b10f944e39829faf16e061fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:31.156170 9797 raft_consensus.cc:3037] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:31.156473 11327 raft_consensus.cc:686] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Running, Role: LEADER
I20240503 08:14:31.156801 11327 consensus_queue.cc:237] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:31.157750 9535 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:31.158833 9797 raft_consensus.cc:2454] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb18b8b10f944e39829faf16e061fd6 in term 1.
I20240503 08:14:31.161234 9389 catalog_manager.cc:5484] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7). New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:31.464769 11347 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.465799 11265 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.476044 11437 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:31.544013 11480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:31.544327 11480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:31.551637 11480 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:31.555461 11347 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.557401 11327 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.561805 11437 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.565462 11437 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.569161 11327 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.572106 11347 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.609475 11265 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.612180 11265 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.618396 11265 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 08:14:31.627980 11327 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.628441 11478 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.637277 11342 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.667434 11520 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:31.678382 11327 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:31.689383 9862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:41805, user_credentials={real_user=slave}} blocked reactor thread for 64379.3us
W20240503 08:14:31.695559 10124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.7:42473, user_credentials={real_user=slave}} blocked reactor thread for 79037.9us
I20240503 08:14:31.701799 11265 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:31.900614 9731 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39353, user_credentials={real_user=slave}} blocked reactor thread for 67405.9us
INSERT report
rows total: 2000
time total: 389.299 ms
time per row: 0.19465 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:14:33.137317 9391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55074:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:33.138059 9391 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_5 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:14:33.147341 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 9428100e4a364805863419b22ca3c682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.147472 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 9428100e4a364805863419b22ca3c682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.147789 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.147897 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.149019 10169 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3941801988415e9fc2620c5ae5aada (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.149399 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3941801988415e9fc2620c5ae5aada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.151297 10170 tablet_service.cc:1467] Processing CreateTablet for tablet 9428100e4a364805863419b22ca3c682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.151647 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.152256 9647 tablet_service.cc:1467] Processing CreateTablet for tablet 43f3607501a84050962dc116210c09cd (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.152624 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.153534 11587 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:33.153971 10038 tablet_service.cc:1467] Processing CreateTablet for tablet 9428100e4a364805863419b22ca3c682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.154347 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.155102 11587 tablet_bootstrap.cc:654] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.155218 10040 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3941801988415e9fc2620c5ae5aada (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.155551 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3941801988415e9fc2620c5ae5aada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.156167 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3941801988415e9fc2620c5ae5aada (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.156570 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3941801988415e9fc2620c5ae5aada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.159206 10039 tablet_service.cc:1467] Processing CreateTablet for tablet 43f3607501a84050962dc116210c09cd (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.159329 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3941801988415e9fc2620c5ae5aada (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.159561 10039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.159669 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3941801988415e9fc2620c5ae5aada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.164386 9907 tablet_service.cc:1467] Processing CreateTablet for tablet 43f3607501a84050962dc116210c09cd (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.164798 9907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.164695 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 43f3607501a84050962dc116210c09cd (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.165138 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.167342 9776 tablet_service.cc:1467] Processing CreateTablet for tablet 9428100e4a364805863419b22ca3c682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.167696 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.169673 10302 tablet_service.cc:1467] Processing CreateTablet for tablet 9428100e4a364805863419b22ca3c682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.170095 10302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.170420 9777 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3941801988415e9fc2620c5ae5aada (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.170774 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3941801988415e9fc2620c5ae5aada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.171231 10300 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3941801988415e9fc2620c5ae5aada (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.171555 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3941801988415e9fc2620c5ae5aada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.174319 10301 tablet_service.cc:1467] Processing CreateTablet for tablet 43f3607501a84050962dc116210c09cd (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.174335 10171 tablet_service.cc:1467] Processing CreateTablet for tablet 43f3607501a84050962dc116210c09cd (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.174650 10301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.174736 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.177613 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 9428100e4a364805863419b22ca3c682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.178083 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.183327 11590 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:33.184422 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 1f3941801988415e9fc2620c5ae5aada (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.184849 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3941801988415e9fc2620c5ae5aada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.185972 11592 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:33.187255 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 43f3607501a84050962dc116210c09cd (DEFAULT_TABLE table=rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:33.187638 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:33.191942 11587 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:33.192116 11587 ts_tablet_manager.cc:1367] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.010s
I20240503 08:14:33.193538 11587 raft_consensus.cc:348] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.194346 11587 raft_consensus.cc:374] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.194574 11587 raft_consensus.cc:729] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.194984 11587 consensus_queue.cc:260] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.196126 11587 ts_tablet_manager.cc:1398] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:33.197165 11591 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:33.197602 11593 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:33.199405 11593 tablet_bootstrap.cc:654] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.200927 11591 tablet_bootstrap.cc:654] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.203572 11590 tablet_bootstrap.cc:654] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.204141 11592 tablet_bootstrap.cc:654] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.218992 11591 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:33.219285 11591 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20240503 08:14:33.220126 11591 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.220513 11591 raft_consensus.cc:374] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.220654 11591 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.220929 11591 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.222401 11591 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:33.222751 11591 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:33.223593 11592 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:33.223804 11592 ts_tablet_manager.cc:1367] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
I20240503 08:14:33.224659 11592 raft_consensus.cc:348] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.225003 11592 raft_consensus.cc:374] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.225185 11592 raft_consensus.cc:729] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.225600 11593 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:33.225499 11592 consensus_queue.cc:260] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.225797 11593 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.003s
I20240503 08:14:33.226418 11592 ts_tablet_manager.cc:1398] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:33.226984 11592 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:33.226902 11593 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.227238 11593 raft_consensus.cc:374] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.227375 11593 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.227670 11593 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.228456 11593 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:14:33.229068 11591 tablet_bootstrap.cc:654] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.239029 11592 tablet_bootstrap.cc:654] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.242295 11587 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:33.244010 11587 tablet_bootstrap.cc:654] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.255831 11593 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:33.257855 11593 tablet_bootstrap.cc:654] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.263259 11598 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:33.266577 11590 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:33.266813 11598 tablet_bootstrap.cc:654] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.266835 11590 ts_tablet_manager.cc:1367] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.010s
I20240503 08:14:33.267406 11589 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:33.268071 11590 raft_consensus.cc:348] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.268507 11590 raft_consensus.cc:374] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.268643 11590 raft_consensus.cc:729] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.269042 11590 consensus_queue.cc:260] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.269770 11590 ts_tablet_manager.cc:1398] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:33.270181 11590 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:33.269500 11589 tablet_bootstrap.cc:654] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.271445 11592 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:33.271965 11590 tablet_bootstrap.cc:654] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.272205 11593 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:33.272404 11593 ts_tablet_manager.cc:1367] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240503 08:14:33.273092 11592 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.012s
I20240503 08:14:33.273295 11593 raft_consensus.cc:348] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.273595 11593 raft_consensus.cc:374] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.273681 11593 raft_consensus.cc:729] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.274230 11592 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.274497 11592 raft_consensus.cc:374] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.274569 11592 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.274976 11592 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.274958 11593 consensus_queue.cc:260] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.275563 11592 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:33.276664 11593 ts_tablet_manager.cc:1398] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:14:33.278064 11593 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:33.278220 11592 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:33.280608 11593 tablet_bootstrap.cc:654] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.281402 11592 tablet_bootstrap.cc:654] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.281675 11587 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:33.281951 11587 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.007s
I20240503 08:14:33.282845 11587 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.283268 11587 raft_consensus.cc:374] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.283484 11587 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.283906 11587 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.287854 11598 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:33.288012 11598 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20240503 08:14:33.288326 11587 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:14:33.288777 11598 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.289023 11598 raft_consensus.cc:374] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.289124 11598 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.289201 11589 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:33.289343 11589 ts_tablet_manager.cc:1367] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240503 08:14:33.289337 11598 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.289868 11598 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:33.290182 11598 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:33.290051 11589 raft_consensus.cc:348] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.290298 11589 raft_consensus.cc:374] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.290374 11589 raft_consensus.cc:729] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.291183 11587 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:33.291272 11589 consensus_queue.cc:260] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.292358 11598 tablet_bootstrap.cc:654] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.293131 11587 tablet_bootstrap.cc:654] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.293740 11591 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:33.293947 11591 ts_tablet_manager.cc:1367] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.014s
I20240503 08:14:33.294111 11590 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:33.294948 11593 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:33.294788 11591 raft_consensus.cc:348] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.295145 11591 raft_consensus.cc:374] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.295221 11591 raft_consensus.cc:729] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.295258 11593 ts_tablet_manager.cc:1367] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240503 08:14:33.295459 11591 consensus_queue.cc:260] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.295769 11590 ts_tablet_manager.cc:1367] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.003s
I20240503 08:14:33.296141 11591 ts_tablet_manager.cc:1398] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:33.296641 11589 ts_tablet_manager.cc:1398] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:14:33.297005 11589 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:33.297859 11590 raft_consensus.cc:348] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.298207 11590 raft_consensus.cc:374] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.298326 11590 raft_consensus.cc:729] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.298717 11590 consensus_queue.cc:260] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.299254 11591 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:33.300434 11590 ts_tablet_manager.cc:1398] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:14:33.300881 11591 tablet_bootstrap.cc:654] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.301829 11593 raft_consensus.cc:348] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.302599 11593 raft_consensus.cc:374] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.302757 11593 raft_consensus.cc:729] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.303113 11593 consensus_queue.cc:260] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.303715 11589 tablet_bootstrap.cc:654] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.303778 11593 ts_tablet_manager.cc:1398] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 08:14:33.305318 11590 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:33.306583 11592 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:33.307292 11592 ts_tablet_manager.cc:1367] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.003s
I20240503 08:14:33.308360 11587 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:33.308562 11587 ts_tablet_manager.cc:1367] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 08:14:33.308643 11592 raft_consensus.cc:348] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.309496 11320 raft_consensus.cc:482] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:33.309635 11590 tablet_bootstrap.cc:654] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.309669 11320 raft_consensus.cc:504] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.310902 11592 raft_consensus.cc:374] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.311002 11320 leader_election.cc:288] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:33.311072 11592 raft_consensus.cc:729] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.311426 11592 consensus_queue.cc:260] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.311930 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:33.311924 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:33.312160 9928 raft_consensus.cc:2454] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.312287 11598 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:33.312851 11592 ts_tablet_manager.cc:1398] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:14:33.313866 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:33.314050 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:33.314256 10321 raft_consensus.cc:2454] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.314323 11598 ts_tablet_manager.cc:1367] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.008s
W20240503 08:14:33.314821 9994 leader_election.cc:341] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:33.315076 11598 raft_consensus.cc:348] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.315454 11598 raft_consensus.cc:374] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.315529 11598 raft_consensus.cc:729] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.312779 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:33.315601 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:33.315677 9664 raft_consensus.cc:2454] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.315815 9796 raft_consensus.cc:2454] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.315790 11598 consensus_queue.cc:260] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.316123 9995 leader_election.cc:302] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters: fa60e870de844325bad4b9185eaee468
I20240503 08:14:33.316385 11598 ts_tablet_manager.cc:1398] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:33.316414 11320 raft_consensus.cc:2781] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:33.316505 11320 raft_consensus.cc:482] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:33.316579 11320 raft_consensus.cc:3037] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.317611 11598 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:33.319183 11598 tablet_bootstrap.cc:654] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.319311 11320 raft_consensus.cc:504] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
W20240503 08:14:33.319950 9993 leader_election.cc:341] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:33.320046 11591 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:33.320179 11591 ts_tablet_manager.cc:1367] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20240503 08:14:33.320688 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:33.320890 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:33.320907 11591 raft_consensus.cc:348] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.321120 9928 raft_consensus.cc:3037] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.321146 11591 raft_consensus.cc:374] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.321238 11591 raft_consensus.cc:729] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.321460 11591 consensus_queue.cc:260] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.324558 9928 raft_consensus.cc:2454] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
W20240503 08:14:33.324993 9994 leader_election.cc:341] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:33.326107 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:33.326303 10321 raft_consensus.cc:3037] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.326314 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
W20240503 08:14:33.326898 9993 leader_election.cc:341] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:33.327271 11591 ts_tablet_manager.cc:1398] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:14:33.327711 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:33.327914 9664 raft_consensus.cc:3037] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.328318 11589 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:33.328480 11589 ts_tablet_manager.cc:1367] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20240503 08:14:33.329178 11589 raft_consensus.cc:348] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.329435 11589 raft_consensus.cc:374] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.329506 11589 raft_consensus.cc:729] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.329754 11589 consensus_queue.cc:260] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.330461 11589 ts_tablet_manager.cc:1398] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:33.330595 9664 raft_consensus.cc:2454] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:33.330780 11589 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:33.331945 11590 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:33.332108 11590 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.004s
I20240503 08:14:33.332317 11589 tablet_bootstrap.cc:654] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:33.332470 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:33.332698 9796 raft_consensus.cc:3037] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.333631 11590 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.333966 11590 raft_consensus.cc:374] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.334095 11590 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.334358 11590 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.335765 11590 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:14:33.336027 9796 raft_consensus.cc:2454] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:33.336481 9993 leader_election.cc:302] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters: b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468
I20240503 08:14:33.336764 11320 leader_election.cc:288] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Requested vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:33.336949 11320 raft_consensus.cc:2781] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:33.337059 11320 raft_consensus.cc:686] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Becoming Leader. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Running, Role: LEADER
I20240503 08:14:33.337431 10321 raft_consensus.cc:2454] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:33.337388 11320 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.338806 11573 raft_consensus.cc:482] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:33.338985 11573 raft_consensus.cc:504] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.340238 11573 leader_election.cc:288] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:33.309334 11587 raft_consensus.cc:348] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.340422 11587 raft_consensus.cc:374] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.340538 11587 raft_consensus.cc:729] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.340648 11589 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:33.340826 11589 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240503 08:14:33.340818 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:33.341015 9796 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:33.341256 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:33.341459 10321 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:33.341470 11589 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.341691 11589 raft_consensus.cc:374] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.341759 11589 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.341986 11589 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.342607 11589 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:14:33.342964 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:33.343187 10059 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:33.343392 11587 consensus_queue.cc:260] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.344558 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:33.344781 10188 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:33.345127 11587 ts_tablet_manager.cc:1398] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20240503 08:14:33.345239 9470 leader_election.cc:302] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5; no voters:
I20240503 08:14:33.345752 11573 raft_consensus.cc:482] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:33.345912 11573 raft_consensus.cc:504] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.346267 11575 raft_consensus.cc:2781] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:33.346376 11575 raft_consensus.cc:482] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:33.346482 11575 raft_consensus.cc:3037] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.346352 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:33.347128 11573 leader_election.cc:288] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:33.347244 9664 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:33.347760 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:33.348129 9929 raft_consensus.cc:2379] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3d3436263d04c57b2d47cc9d766efa5 in current term 1: Already voted for candidate a1402a75453d4ccf992606dcce77dc51 in this term.
I20240503 08:14:33.348654 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:33.348654 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:33.348901 10321 raft_consensus.cc:2379] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3d3436263d04c57b2d47cc9d766efa5 in current term 1: Already voted for candidate a1402a75453d4ccf992606dcce77dc51 in this term.
I20240503 08:14:33.348894 9796 raft_consensus.cc:2379] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3d3436263d04c57b2d47cc9d766efa5 in current term 1: Already voted for candidate a1402a75453d4ccf992606dcce77dc51 in this term.
I20240503 08:14:33.349068 11575 raft_consensus.cc:504] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.349794 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:33.350003 10188 raft_consensus.cc:2454] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:33.350476 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:33.350705 9664 raft_consensus.cc:2379] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3d3436263d04c57b2d47cc9d766efa5 in current term 1: Already voted for candidate a1402a75453d4ccf992606dcce77dc51 in this term.
I20240503 08:14:33.350843 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f3607501a84050962dc116210c09cd" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:33.351152 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:33.351163 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:33.351333 9796 raft_consensus.cc:3037] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.351341 9928 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:33.351058 9391 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1402a75453d4ccf992606dcce77dc51 (127.24.123.5). New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:33.352993 11598 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:33.353104 11598 ts_tablet_manager.cc:1367] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.002s
I20240503 08:14:33.353545 9796 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:33.353760 11598 raft_consensus.cc:348] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.354074 11598 raft_consensus.cc:374] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:33.354213 11598 raft_consensus.cc:729] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:33.354184 9470 leader_election.cc:302] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468; no voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51
I20240503 08:14:33.354549 11598 consensus_queue.cc:260] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.354955 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:33.355147 9664 raft_consensus.cc:3037] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.355372 11598 ts_tablet_manager.cc:1398] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:33.355950 11573 raft_consensus.cc:3037] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.357277 9664 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:33.358187 11573 raft_consensus.cc:2726] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:33.358693 11575 leader_election.cc:288] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Requested vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:33.359133 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:33.359301 10059 raft_consensus.cc:3037] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.359944 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:33.360145 10321 raft_consensus.cc:3037] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.360110 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:33.360318 10188 raft_consensus.cc:3037] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.361783 10059 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:33.362164 9470 leader_election.cc:302] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:33.362192 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9428100e4a364805863419b22ca3c682" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:33.362358 9928 raft_consensus.cc:3037] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.362529 11575 raft_consensus.cc:2781] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:33.362658 11575 raft_consensus.cc:686] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Becoming Leader. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Running, Role: LEADER
I20240503 08:14:33.362740 10321 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:33.362810 10188 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:33.363143 11575 consensus_queue.cc:237] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:33.364418 9928 raft_consensus.cc:2454] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:33.366470 9391 catalog_manager.cc:5484] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:33.398519 11320 raft_consensus.cc:482] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:33.398712 11320 raft_consensus.cc:504] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.400192 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:33.400429 9533 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.400435 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:33.400632 10188 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.401250 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:33.400202 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:33.401466 9664 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.401648 10321 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.401988 11320 leader_election.cc:288] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:33.402071 9993 leader_election.cc:302] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:33.402361 11320 raft_consensus.cc:2781] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:33.402453 11320 raft_consensus.cc:482] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:33.402544 11320 raft_consensus.cc:3037] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.402724 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:33.402940 9796 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.403127 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:33.403376 9928 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:33.405239 11320 raft_consensus.cc:504] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.406333 11320 leader_election.cc:288] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Requested vote from peers fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:33.406319 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:33.406497 10188 raft_consensus.cc:3037] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.406849 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:33.407076 9796 raft_consensus.cc:3037] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.407263 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:33.407480 10321 raft_consensus.cc:3037] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.408159 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:33.408357 9533 raft_consensus.cc:3037] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.408504 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:33.408739 9928 raft_consensus.cc:3037] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.409076 10188 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:33.409925 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f3941801988415e9fc2620c5ae5aada" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:33.410132 9664 raft_consensus.cc:3037] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.410744 9796 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:33.410974 9533 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:33.411294 9993 leader_election.cc:302] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:33.411548 11320 raft_consensus.cc:2781] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:33.411702 11320 raft_consensus.cc:686] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Becoming Leader. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Running, Role: LEADER
I20240503 08:14:33.412012 11320 consensus_queue.cc:237] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:33.412757 9664 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:33.413424 10321 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:33.415112 9928 raft_consensus.cc:2454] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:33.415866 9391 catalog_manager.cc:5484] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1402a75453d4ccf992606dcce77dc51 (127.24.123.5). New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:33.749909 11320 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.765892 11575 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:33.801630 11620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:33.802007 11620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:33.809662 11620 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:33.820672 10188 raft_consensus.cc:3037] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:33.824220 11320 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.834661 11320 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.837162 11320 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.856379 11614 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240503 08:14:33.863126 11575 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.872651 11575 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.882548 11614 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.888320 11575 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.911067 11614 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.913993 11614 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.941144 11648 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.945822 11644 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.965037 11665 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.967095 11669 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.998175 11646 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:33.998674 11646 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 411.972 ms
time per row: 0.205986 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:14:35.435498 9385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55074:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:14:35.436223 9385 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_6 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:14:35.445854 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.445938 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.446331 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.446390 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.446379 10039 tablet_service.cc:1467] Processing CreateTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.446805 10039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.450067 10170 tablet_service.cc:1467] Processing CreateTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.450469 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.450627 9776 tablet_service.cc:1467] Processing CreateTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.450775 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb26d36a69d4f4ab43401822096756f (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.451241 10300 tablet_service.cc:1467] Processing CreateTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.451645 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.452474 9777 tablet_service.cc:1467] Processing CreateTablet for tablet 7247b6f7e08946709b215d2f6a26a2a4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.453033 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.455237 10169 tablet_service.cc:1467] Processing CreateTablet for tablet 7247b6f7e08946709b215d2f6a26a2a4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.455320 10302 tablet_service.cc:1467] Processing CreateTablet for tablet 7247b6f7e08946709b215d2f6a26a2a4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.455617 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7247b6f7e08946709b215d2f6a26a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.455721 10302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7247b6f7e08946709b215d2f6a26a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.459203 10171 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb26d36a69d4f4ab43401822096756f (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.459590 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb26d36a69d4f4ab43401822096756f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.459579 10301 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb26d36a69d4f4ab43401822096756f (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.459977 10301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb26d36a69d4f4ab43401822096756f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.463754 9907 tablet_service.cc:1467] Processing CreateTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.464212 9907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.464613 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb26d36a69d4f4ab43401822096756f (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.465013 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb26d36a69d4f4ab43401822096756f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.467679 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 7247b6f7e08946709b215d2f6a26a2a4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.468135 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7247b6f7e08946709b215d2f6a26a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.468228 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb26d36a69d4f4ab43401822096756f (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.468622 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb26d36a69d4f4ab43401822096756f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.471000 10040 tablet_service.cc:1467] Processing CreateTablet for tablet 7247b6f7e08946709b215d2f6a26a2a4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.471299 9647 tablet_service.cc:1467] Processing CreateTablet for tablet 7247b6f7e08946709b215d2f6a26a2a4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.471426 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7247b6f7e08946709b215d2f6a26a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.471637 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7247b6f7e08946709b215d2f6a26a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.475242 10038 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb26d36a69d4f4ab43401822096756f (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.475623 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb26d36a69d4f4ab43401822096756f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.475565 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 7247b6f7e08946709b215d2f6a26a2a4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.476004 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7247b6f7e08946709b215d2f6a26a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.477062 11741 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:35.478452 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb26d36a69d4f4ab43401822096756f (DEFAULT_TABLE table=rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:35.478741 11741 tablet_bootstrap.cc:654] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.478812 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb26d36a69d4f4ab43401822096756f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.481354 11741 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:35.481505 11741 ts_tablet_manager.cc:1367] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:14:35.482295 11741 raft_consensus.cc:348] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.482566 11741 raft_consensus.cc:374] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.482643 11741 raft_consensus.cc:729] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.482944 11741 consensus_queue.cc:260] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.483752 11741 ts_tablet_manager.cc:1398] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:35.484153 11741 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:35.485821 11741 tablet_bootstrap.cc:654] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.493983 11741 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:35.494227 11741 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20240503 08:14:35.495039 11741 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.495379 11741 raft_consensus.cc:374] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.495467 11741 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.495755 11741 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.496477 11741 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:35.496806 11741 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6: Bootstrap starting.
I20240503 08:14:35.498610 11741 tablet_bootstrap.cc:654] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.516350 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7247b6f7e08946709b215d2f6a26a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.519097 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb26d36a69d4f4ab43401822096756f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:35.529508 11746 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:35.531977 11747 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:35.535014 11748 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:35.538830 11744 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:35.538985 11748 tablet_bootstrap.cc:654] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.540517 11744 tablet_bootstrap.cc:654] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.542779 11749 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:35.543761 11740 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:35.545895 11740 tablet_bootstrap.cc:654] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.546268 11749 tablet_bootstrap.cc:654] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.549170 11746 tablet_bootstrap.cc:654] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.550719 11747 tablet_bootstrap.cc:654] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.552016 11740 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:35.552516 11746 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:35.552701 11740 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240503 08:14:35.552990 11746 ts_tablet_manager.cc:1367] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240503 08:14:35.553129 11749 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:35.553280 11749 ts_tablet_manager.cc:1367] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 08:14:35.553665 11740 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.553979 11746 raft_consensus.cc:348] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.554360 11746 raft_consensus.cc:374] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.554489 11746 raft_consensus.cc:729] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.554967 11740 raft_consensus.cc:374] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.554783 11746 consensus_queue.cc:260] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.555270 11740 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.555765 11740 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.556358 11746 ts_tablet_manager.cc:1398] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:35.556679 11746 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:35.558036 11740 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:14:35.558494 11746 tablet_bootstrap.cc:654] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.559350 11740 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:35.563412 11740 tablet_bootstrap.cc:654] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.568779 11746 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:35.568966 11746 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240503 08:14:35.571705 11746 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.572026 11746 raft_consensus.cc:374] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.572103 11746 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.572417 11746 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.573323 11740 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:35.573485 11747 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:35.573894 11740 ts_tablet_manager.cc:1367] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20240503 08:14:35.573997 11746 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 08:14:35.574330 11746 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51: Bootstrap starting.
I20240503 08:14:35.575433 11740 raft_consensus.cc:348] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.576072 11746 tablet_bootstrap.cc:654] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.576422 11740 raft_consensus.cc:374] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.576685 11740 raft_consensus.cc:729] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.577164 11740 consensus_queue.cc:260] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.577751 11747 ts_tablet_manager.cc:1367] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240503 08:14:35.578958 11740 ts_tablet_manager.cc:1398] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:14:35.579684 11740 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:35.583081 11740 tablet_bootstrap.cc:654] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.585361 11740 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5: No bootstrap required, opened a new log
I20240503 08:14:35.585548 11740 ts_tablet_manager.cc:1367] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:14:35.586403 11744 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:35.586606 11744 ts_tablet_manager.cc:1367] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.009s
I20240503 08:14:35.586756 11749 raft_consensus.cc:348] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.587054 11749 raft_consensus.cc:374] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.587155 11749 raft_consensus.cc:729] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.586388 11747 raft_consensus.cc:348] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.587502 11747 raft_consensus.cc:374] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.587684 11747 raft_consensus.cc:729] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.587697 11744 raft_consensus.cc:348] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.588053 11744 raft_consensus.cc:374] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.587956 11749 consensus_queue.cc:260] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.588173 11744 raft_consensus.cc:729] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.588505 11744 consensus_queue.cc:260] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.590413 11744 ts_tablet_manager.cc:1398] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:14:35.590752 11744 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:35.593283 11744 tablet_bootstrap.cc:654] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.592924 11740 raft_consensus.cc:348] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.594380 11740 raft_consensus.cc:374] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.594516 11740 raft_consensus.cc:729] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.594808 11740 consensus_queue.cc:260] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.595556 11740 ts_tablet_manager.cc:1398] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:14:35.600646 11744 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:35.600868 11744 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240503 08:14:35.601738 11744 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.602063 11744 raft_consensus.cc:374] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.602222 11744 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.602885 11744 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.603675 11744 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:35.604020 11744 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:35.605569 11744 tablet_bootstrap.cc:654] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.607666 11749 ts_tablet_manager.cc:1398] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.054s user 0.002s sys 0.000s
I20240503 08:14:35.588371 11747 consensus_queue.cc:260] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.608505 11747 ts_tablet_manager.cc:1398] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240503 08:14:35.615221 11744 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542: No bootstrap required, opened a new log
I20240503 08:14:35.615401 11744 ts_tablet_manager.cc:1367] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 08:14:35.616135 11744 raft_consensus.cc:348] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.616447 11744 raft_consensus.cc:374] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.616720 11744 raft_consensus.cc:729] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.617033 11744 consensus_queue.cc:260] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.618721 11744 ts_tablet_manager.cc:1398] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:35.640319 11712 raft_consensus.cc:482] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:35.640477 11712 raft_consensus.cc:504] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.642028 11712 leader_election.cc:288] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:35.641948 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:35.642637 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:35.643586 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
W20240503 08:14:35.643592 9469 leader_election.cc:341] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:35.643854 9929 raft_consensus.cc:2454] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
W20240503 08:14:35.644085 9470 leader_election.cc:341] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:35.644738 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:35.645000 10058 raft_consensus.cc:2454] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:35.646201 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:35.646346 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:35.646597 10321 raft_consensus.cc:2454] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:35.647234 9468 leader_election.cc:302] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5; no voters: d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468
W20240503 08:14:35.647514 9468 leader_election.cc:341] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:35.647874 11712 raft_consensus.cc:2781] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:35.647980 11712 raft_consensus.cc:482] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:35.648098 11712 raft_consensus.cc:3037] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.650975 11712 raft_consensus.cc:504] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.652441 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:35.652464 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:35.652698 10058 raft_consensus.cc:3037] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:14:35.654655 9470 leader_election.cc:341] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:35.655570 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:35.656008 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
W20240503 08:14:35.656121 9469 leader_election.cc:341] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:35.656231 9929 raft_consensus.cc:3037] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.658210 9929 raft_consensus.cc:2454] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:35.658679 11712 leader_election.cc:288] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Requested vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:35.659305 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:35.659425 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:35.659648 10321 raft_consensus.cc:3037] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.661929 10321 raft_consensus.cc:2454] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
W20240503 08:14:35.663053 9468 leader_election.cc:341] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:35.663811 10058 raft_consensus.cc:2454] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:35.664341 9470 leader_election.cc:302] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5; no voters: 8de08e679a904ee48ba659861c730725, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468
I20240503 08:14:35.670476 11712 raft_consensus.cc:2781] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:35.670614 11712 raft_consensus.cc:686] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Becoming Leader. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Running, Role: LEADER
I20240503 08:14:35.672502 11712 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.676342 11749 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:35.678037 11749 tablet_bootstrap.cc:654] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.680400 11741 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6: No bootstrap required, opened a new log
I20240503 08:14:35.680588 11741 ts_tablet_manager.cc:1367] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6: Time spent bootstrapping tablet: real 0.184s user 0.079s sys 0.096s
I20240503 08:14:35.680665 11747 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:35.681455 11741 raft_consensus.cc:348] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.681728 11741 raft_consensus.cc:374] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.681838 11741 raft_consensus.cc:729] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb18b8b10f944e39829faf16e061fd6, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.682123 11741 consensus_queue.cc:260] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.682349 11747 tablet_bootstrap.cc:654] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.682752 11741 ts_tablet_manager.cc:1398] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:35.685495 11748 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:35.685716 11748 ts_tablet_manager.cc:1367] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.000s
I20240503 08:14:35.687948 11748 raft_consensus.cc:348] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.688225 11748 raft_consensus.cc:374] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.689550 11748 raft_consensus.cc:729] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.689899 11748 consensus_queue.cc:260] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.690593 11748 ts_tablet_manager.cc:1398] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:14:35.690984 11748 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:35.699047 11748 tablet_bootstrap.cc:654] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.701942 11749 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:35.702978 9385 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:35.704133 11749 ts_tablet_manager.cc:1367] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240503 08:14:35.705092 11749 raft_consensus.cc:348] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.710341 11749 raft_consensus.cc:374] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.714505 11764 raft_consensus.cc:482] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:35.714771 11764 raft_consensus.cc:504] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.716260 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:35.716524 9666 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:35.716720 11748 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:35.716881 11748 ts_tablet_manager.cc:1367] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.008s
I20240503 08:14:35.718513 11764 leader_election.cc:288] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:35.717635 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:35.718688 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:35.718914 10058 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:35.718955 11749 raft_consensus.cc:729] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.719589 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:35.719485 11749 consensus_queue.cc:260] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.719807 10321 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:35.719954 9796 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:35.721295 11749 ts_tablet_manager.cc:1398] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240503 08:14:35.722656 9468 leader_election.cc:302] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:35.723086 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:35.723332 11747 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:35.723542 11747 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
W20240503 08:14:35.724267 9469 leader_election.cc:341] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:35.724694 11765 raft_consensus.cc:2781] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:35.717629 11748 raft_consensus.cc:348] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.725018 11746 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51: No bootstrap required, opened a new log
I20240503 08:14:35.725190 11746 ts_tablet_manager.cc:1367] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51: Time spent bootstrapping tablet: real 0.151s user 0.052s sys 0.083s
I20240503 08:14:35.725342 11765 raft_consensus.cc:482] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:35.725075 11748 raft_consensus.cc:374] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.725678 11748 raft_consensus.cc:729] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.725961 11746 raft_consensus.cc:348] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.726068 11748 consensus_queue.cc:260] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.726303 11746 raft_consensus.cc:374] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.726395 11746 raft_consensus.cc:729] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.726632 11746 consensus_queue.cc:260] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.726855 11748 ts_tablet_manager.cc:1398] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:14:35.727260 11748 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1: Bootstrap starting.
I20240503 08:14:35.728060 11746 ts_tablet_manager.cc:1398] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:14:35.729887 11748 tablet_bootstrap.cc:654] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.739686 11765 raft_consensus.cc:3037] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.725131 11747 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.740990 11747 raft_consensus.cc:374] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.741128 11747 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.741127 11748 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1: No bootstrap required, opened a new log
I20240503 08:14:35.741336 11748 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20240503 08:14:35.741429 11747 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.742102 11747 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20240503 08:14:35.742317 11748 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.742659 11748 raft_consensus.cc:374] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.742769 11748 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1eb5e69c6264a20a6e8297cc5deb8d1, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.743089 11748 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.743844 11748 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:35.746694 11663 raft_consensus.cc:482] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:35.746902 11663 raft_consensus.cc:504] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.748483 11663 leader_election.cc:288] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:35.749334 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:35.749572 9796 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:35.750706 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:35.750968 10321 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:35.751196 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:35.751464 9533 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:35.752501 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:35.752795 9666 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:35.753202 9995 leader_election.cc:302] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0cb18b8b10f944e39829faf16e061fd6, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1; no voters:
I20240503 08:14:35.753264 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:35.753472 9929 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:35.753573 11663 raft_consensus.cc:2781] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:35.753687 11663 raft_consensus.cc:482] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:35.753767 11663 raft_consensus.cc:3037] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.753772 11749 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:35.754557 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468" is_pre_election: true
I20240503 08:14:35.754961 10189 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 0.
I20240503 08:14:35.756583 11663 raft_consensus.cc:504] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.758358 11747 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:35.758945 11663 leader_election.cc:288] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Requested vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:35.759126 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:35.759404 10189 raft_consensus.cc:3037] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.759392 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:35.759445 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:35.759680 9533 raft_consensus.cc:3037] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.759677 9664 raft_consensus.cc:3037] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.762395 10189 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:35.762617 9664 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:35.762624 9533 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:35.763656 9993 leader_election.cc:302] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:35.764125 10321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:35.764353 10321 raft_consensus.cc:3037] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.765417 11749 tablet_bootstrap.cc:654] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.765908 11765 raft_consensus.cc:504] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.767603 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:35.767879 9796 raft_consensus.cc:3037] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.769068 11747 tablet_bootstrap.cc:654] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:35.770040 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:35.770190 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:35.770385 9929 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 0.
I20240503 08:14:35.771106 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:35.771296 9929 raft_consensus.cc:3037] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.772526 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:35.772967 9927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542"
I20240503 08:14:35.773206 9927 raft_consensus.cc:3037] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.774416 9929 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:35.776005 11773 raft_consensus.cc:2781] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:35.776026 9927 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:35.776952 11765 leader_election.cc:288] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Requested vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
W20240503 08:14:35.777524 9469 leader_election.cc:341] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:14:35.779538 11773 raft_consensus.cc:686] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Becoming Leader. State: Replica: a1402a75453d4ccf992606dcce77dc51, State: Running, Role: LEADER
I20240503 08:14:35.779934 11773 consensus_queue.cc:237] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.782735 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:35.782965 10058 raft_consensus.cc:3037] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.785426 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7247b6f7e08946709b215d2f6a26a2a4" candidate_uuid: "a1402a75453d4ccf992606dcce77dc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:35.785670 9797 raft_consensus.cc:3037] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.786129 10058 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:35.786799 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "b3d3436263d04c57b2d47cc9d766efa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:35.787056 9665 raft_consensus.cc:3037] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.788801 9796 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:35.788821 10320 raft_consensus.cc:3037] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.789588 9665 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:35.790217 9797 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:35.790843 9390 catalog_manager.cc:5484] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1402a75453d4ccf992606dcce77dc51 (127.24.123.5). New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:35.791594 11749 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: No bootstrap required, opened a new log
I20240503 08:14:35.791801 11749 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.005s
I20240503 08:14:35.791915 10320 raft_consensus.cc:2454] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3d3436263d04c57b2d47cc9d766efa5 in term 1.
I20240503 08:14:35.792044 9470 leader_election.cc:302] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5; no voters: fa60e870de844325bad4b9185eaee468
I20240503 08:14:35.792512 11762 raft_consensus.cc:2781] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:35.792703 11762 raft_consensus.cc:686] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Becoming Leader. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Running, Role: LEADER
I20240503 08:14:35.793084 11762 consensus_queue.cc:237] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.793421 10321 raft_consensus.cc:2454] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1402a75453d4ccf992606dcce77dc51 in term 1.
I20240503 08:14:35.793685 11747 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468: No bootstrap required, opened a new log
I20240503 08:14:35.793875 11747 ts_tablet_manager.cc:1367] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.007s
I20240503 08:14:35.794723 11747 raft_consensus.cc:348] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.795042 11747 raft_consensus.cc:374] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.795177 11747 raft_consensus.cc:729] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.795421 11747 consensus_queue.cc:260] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.796075 11747 ts_tablet_manager.cc:1398] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:35.796808 11749 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.797080 11749 raft_consensus.cc:374] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:35.797186 11749 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: FOLLOWER
I20240503 08:14:35.797427 11749 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.798023 11749 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:14:35.797930 9390 catalog_manager.cc:5484] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1). New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:35.805694 11734 raft_consensus.cc:482] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:35.805867 11734 raft_consensus.cc:504] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.807272 11734 leader_election.cc:288] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:35.807942 9927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:35.807979 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:35.808265 9927 raft_consensus.cc:2379] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:35.808431 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:35.808722 10059 raft_consensus.cc:2379] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:35.809252 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:35.809474 9797 raft_consensus.cc:2379] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:35.809844 10124 leader_election.cc:302] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: fa60e870de844325bad4b9185eaee468; no voters: 06f1b17731834350a289de93abab8542, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5
I20240503 08:14:35.810101 11734 raft_consensus.cc:3037] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.810035 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:35.810277 10320 raft_consensus.cc:2379] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:35.811440 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cb26d36a69d4f4ab43401822096756f" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:35.812131 9664 raft_consensus.cc:2379] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:35.812417 11734 raft_consensus.cc:2726] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:35.826328 11734 raft_consensus.cc:482] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:35.826592 11734 raft_consensus.cc:504] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:35.828006 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" is_pre_election: true
I20240503 08:14:35.828197 9664 raft_consensus.cc:2454] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:35.828368 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5" is_pre_election: true
I20240503 08:14:35.828953 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de08e679a904ee48ba659861c730725" is_pre_election: true
I20240503 08:14:35.829169 9797 raft_consensus.cc:2454] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa60e870de844325bad4b9185eaee468 in term 0.
I20240503 08:14:35.829494 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1402a75453d4ccf992606dcce77dc51" is_pre_election: true
I20240503 08:14:35.829756 10059 raft_consensus.cc:2379] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:35.829844 9927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f1b17731834350a289de93abab8542" is_pre_election: true
I20240503 08:14:35.829927 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8" candidate_uuid: "fa60e870de844325bad4b9185eaee468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb18b8b10f944e39829faf16e061fd6" is_pre_election: true
I20240503 08:14:35.830204 10320 raft_consensus.cc:2379] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:35.830205 9927 raft_consensus.cc:2379] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa60e870de844325bad4b9185eaee468 in current term 1: Already voted for candidate b3d3436263d04c57b2d47cc9d766efa5 in this term.
I20240503 08:14:35.830734 11734 leader_election.cc:288] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 06f1b17731834350a289de93abab8542 (127.24.123.4:38565), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:35.831019 10126 leader_election.cc:302] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: fa60e870de844325bad4b9185eaee468; no voters: 06f1b17731834350a289de93abab8542, 0cb18b8b10f944e39829faf16e061fd6, a1402a75453d4ccf992606dcce77dc51, b3d3436263d04c57b2d47cc9d766efa5
I20240503 08:14:35.831430 11734 raft_consensus.cc:3037] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:35.833894 11734 raft_consensus.cc:2726] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:14:36.084338 9664 raft_consensus.cc:3037] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:36.088367 11762 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.134593 9797 raft_consensus.cc:3037] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:36.137271 11762 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.142580 11762 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.148435 11762 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.166554 11764 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.169389 11663 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.170660 11763 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.180325 11663 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.197903 11764 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.205093 11764 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:36.216531 11780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:36.217000 11780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:36.228142 11780 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:14:36.231168 11763 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.264844 11765 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.269227 11762 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.279127 11762 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.290866 11663 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.312263 11775 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.323133 11775 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:36.323511 11663 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 464.088 ms
time per row: 0.232044 ms
I20240503 08:14:36.958721 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.9:44619
--local_ip_for_outbound_sockets=127.24.123.9
--tserver_master_addrs=127.24.123.62:44669
--webserver_port=43553
--webserver_interface=127.24.123.9
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:14:37.323568 11874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:37.323873 11874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:37.324009 11874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:37.332098 11874 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:37.332232 11874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:37.332374 11874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.9
I20240503 08:14:37.341603 11874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.9:44619
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.24.123.9
--webserver_port=43553
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.9
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:37.342811 11874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:37.344703 11874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:37.357080 11882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:37.357580 11881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:37.358065 11884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:37.358712 11874 server_base.cc:1034] running on GCE node
I20240503 08:14:37.359329 11874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:37.360348 11874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:37.361549 11874 hybrid_clock.cc:648] HybridClock initialized: now 1714724077361500 us; error 47 us; skew 500 ppm
I20240503 08:14:37.361909 11874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:37.363999 11874 webserver.cc:460] Webserver started at http://127.24.123.9:43553/ using document root <none> and password file <none>
I20240503 08:14:37.364655 11874 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:37.364773 11874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:37.370184 11874 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:14:37.373057 11890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:37.374979 11874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 08:14:37.375136 11874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "3a70e21b51a142bd8f7dc93fa58742fb"
format_stamp: "Formatted at 2024-05-03 08:14:17 on dist-test-slave-gkrw"
I20240503 08:14:37.376209 11874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:37.395709 11874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:37.396615 11874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:37.396907 11874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:37.397938 11874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:37.399947 11874 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:37.400031 11874 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:37.400149 11874 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:37.400209 11874 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:37.451748 11874 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.9:44619
I20240503 08:14:37.451912 12002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.9:44619 every 8 connection(s)
I20240503 08:14:37.453387 11874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 08:14:37.459453 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 11874
I20240503 08:14:37.460186 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.10:33439
--local_ip_for_outbound_sockets=127.24.123.10
--tserver_master_addrs=127.24.123.62:44669
--webserver_port=39305
--webserver_interface=127.24.123.10
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:14:37.469887 12003 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:37.470243 12003 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:37.471181 12003 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:37.473456 9392 ts_manager.cc:194] Re-registered known tserver with Master: 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619)
I20240503 08:14:37.474673 9392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.9:45331
W20240503 08:14:37.814169 12008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:37.814433 12008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:37.814555 12008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:37.822321 12008 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:37.822448 12008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:37.822619 12008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.10
I20240503 08:14:37.836463 12008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.10:33439
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.24.123.10
--webserver_port=39305
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.10
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:37.837893 12008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:37.839895 12008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:37.857287 12013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:37.858045 12014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:37.858457 12016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:37.859452 12008 server_base.cc:1034] running on GCE node
I20240503 08:14:37.860082 12008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:37.860977 12008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:37.862195 12008 hybrid_clock.cc:648] HybridClock initialized: now 1714724077862127 us; error 50 us; skew 500 ppm
I20240503 08:14:37.862594 12008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:37.865203 12008 webserver.cc:460] Webserver started at http://127.24.123.10:39305/ using document root <none> and password file <none>
I20240503 08:14:37.866125 12008 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:37.866312 12008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:37.872094 12008 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:14:37.875103 12022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:37.877143 12008 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20240503 08:14:37.877290 12008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "925c339b047b4cf5bae840c5a357d3de"
format_stamp: "Formatted at 2024-05-03 08:14:18 on dist-test-slave-gkrw"
I20240503 08:14:37.878041 12008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:37.903950 12008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:37.904735 12008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:37.905001 12008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:37.905898 12008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:37.907408 12008 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:37.907487 12008 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:37.907603 12008 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:37.907651 12008 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:37.937853 12008 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.10:33439
I20240503 08:14:37.937903 12134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.10:33439 every 8 connection(s)
I20240503 08:14:37.939651 12008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 08:14:37.942735 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 12008
I20240503 08:14:37.951884 12135 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:37.952152 12135 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:37.952971 12135 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:37.954453 9392 ts_manager.cc:194] Re-registered known tserver with Master: 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439)
I20240503 08:14:37.955317 9392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.10:33821
I20240503 08:14:37.984321 25068 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEUWrla/build/asan/bin/kudu
/tmp/dist-test-taskEUWrla/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.123.8:43883
--local_ip_for_outbound_sockets=127.24.123.8
--tserver_master_addrs=127.24.123.62:44669
--webserver_port=41085
--webserver_interface=127.24.123.8
--builtin_ntp_servers=127.24.123.20:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:14:38.335366 12138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:38.335700 12138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:38.335860 12138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:38.343844 12138 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:14:38.343948 12138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:38.344110 12138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.123.8
I20240503 08:14:38.355279 12138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.123.20:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.123.8:43883
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.123.8
--webserver_port=41085
--tserver_master_addrs=127.24.123.62:44669
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.123.8
--log_dir=/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:14:38.357028 12138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:38.358774 12138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:38.372953 12150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:38.373270 12147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:38.373625 12146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:38.374605 12138 server_base.cc:1034] running on GCE node
I20240503 08:14:38.375201 12138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:38.376003 12138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:38.377192 12138 hybrid_clock.cc:648] HybridClock initialized: now 1714724078377144 us; error 43 us; skew 500 ppm
I20240503 08:14:38.377472 12138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:14:38.380389 12138 webserver.cc:460] Webserver started at http://127.24.123.8:41085/ using document root <none> and password file <none>
I20240503 08:14:38.381080 12138 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:38.381203 12138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:38.386673 12138 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 08:14:38.389631 12156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:38.391139 12138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:38.391295 12138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ca5fa388d5b84a0abac1c05275155dcf"
format_stamp: "Formatted at 2024-05-03 08:14:17 on dist-test-slave-gkrw"
I20240503 08:14:38.392299 12138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:38.423864 12138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:38.424623 12138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:38.424872 12138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:38.425756 12138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:38.427222 12138 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:38.427294 12138 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:38.427398 12138 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:38.427459 12138 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:38.471505 12138 rpc_server.cc:307] RPC server started. Bound to: 127.24.123.8:43883
I20240503 08:14:38.471577 12268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.123.8:43883 every 8 connection(s)
I20240503 08:14:38.473201 12138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:14:38.474540 25068 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEUWrla/build/asan/bin/kudu as pid 12138
I20240503 08:14:38.477393 12003 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
I20240503 08:14:38.495327 12269 heartbeater.cc:347] Connected to a master server at 127.24.123.62:44669
I20240503 08:14:38.495654 12269 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:38.496466 12269 heartbeater.cc:511] Master 127.24.123.62:44669 requested a full tablet report, sending...
I20240503 08:14:38.498147 9392 ts_manager.cc:194] Re-registered known tserver with Master: ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883)
I20240503 08:14:38.499190 9392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.123.8:36835
I20240503 08:14:38.957731 12135 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
I20240503 08:14:39.026542 9909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:39.041328 12070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.047461 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:14:39.065383 10038 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:39.093398 9777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:39.096741 11938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.157060 9513 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:39.169445 9646 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:39.178028 10169 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:39.212790 12204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.431237 9909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:39.444466 12070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.482342 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:14:39.482478 11938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.484267 9777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:39.494509 10038 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:39.501538 12269 heartbeater.cc:503] Master 127.24.123.62:44669 was elected leader, sending a full tablet report...
I20240503 08:14:39.522475 12204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.522627 9646 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:39.552978 10169 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:39.559386 9513 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:39.813822 9909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:39.863540 12070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.865881 11938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.887982 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:14:39.895185 10038 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:39.905426 9646 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:39.906301 9513 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:39.910298 9777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:39.918226 12204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:39.949304 10169 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:40.058261 9797 consensus_queue.cc:237] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:40.061968 9533 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.062206 9927 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.062469 10058 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.062754 12391 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.062925 9666 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.062793 10188 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.063617 12391 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.063884 10320 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.064180 12391 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.064733 12391 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.065229 11733 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.071115 12391 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.086927 12424 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.087723 9664 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.090258 9533 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.091236 10059 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.094142 9927 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.097657 9393 catalog_manager.cc:5484] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.103722 10188 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.105932 10320 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.119104 9797 consensus_queue.cc:237] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:40.124487 10188 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.124555 10059 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.124795 10320 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.125337 9927 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.125391 12391 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.125948 12391 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.126205 12148 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.127616 11733 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.130215 12148 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.130833 9927 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
W20240503 08:14:40.132041 9731 consensus_peers.cc:485] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f13c7cfe35b3403f87940a124e6d9cda. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.133152 10320 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.134390 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f13c7cfe35b3403f87940a124e6d9cda with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:40.136108 10188 consensus_queue.cc:237] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:40.136564 9385 catalog_manager.cc:5484] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.138990 9666 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.139684 9533 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.139783 12391 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.140594 12424 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.140622 10191 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.142168 9533 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.143659 10058 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.143708 9927 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.144706 11875 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.145256 11875 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.148530 10321 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.151994 10059 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.156019 12328 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.156430 9797 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.161916 12328 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.164568 9666 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.171697 12328 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.175180 9929 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.183079 9389 catalog_manager.cc:5484] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.195382 10320 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.215782 9666 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.216346 9533 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.219136 11875 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.219692 11875 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.226792 10059 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.233664 9797 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.239244 10188 consensus_queue.cc:237] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:40.244361 9929 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.244580 10059 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.244939 11875 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.245173 12328 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.257503 10320 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.257896 9796 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.258713 12450 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.259357 12450 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.261739 9533 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.262069 12328 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.265136 9379 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cc9f0afdc01d4d03917b9040c2053426 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:40.265551 9797 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.265901 9666 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.265866 10059 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.268515 9389 catalog_manager.cc:5484] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:40.271540 9535 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.271549 9929 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.272735 12328 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.273128 10320 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.275384 9535 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
W20240503 08:14:40.280135 10124 consensus_peers.cc:485] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc9f0afdc01d4d03917b9040c2053426. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.287061 9664 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.288437 11875 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.291738 9666 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.312536 10059 consensus_queue.cc:237] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:40.315742 9666 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.315891 9797 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.316459 11879 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.316556 9533 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.317091 11879 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.318017 12319 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.318426 10320 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.319643 9929 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.321887 12319 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.322364 12319 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.324443 10188 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.325168 12319 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.326092 12458 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.326654 9929 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.331974 9392 catalog_manager.cc:5484] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.333240 10320 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.343263 9666 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.351712 9796 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.355994 9533 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.359107 10188 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.363322 10059 consensus_queue.cc:237] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:40.367842 10188 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.368795 9666 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.369125 9535 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.369314 10320 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.369239 11879 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.369436 9929 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.369989 11879 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.370270 12459 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.370548 12319 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.370790 12458 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.371135 9796 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.371572 9533 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:40.375428 12458 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.376049 9666 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.381050 9380 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1f3941801988415e9fc2620c5ae5aada with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:40.381336 10188 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.383131 9392 catalog_manager.cc:5484] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:40.385408 12459 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.385550 9927 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.396198 9796 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.398687 9535 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.401007 10321 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
W20240503 08:14:40.403152 9993 consensus_peers.cc:485] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f3941801988415e9fc2620c5ae5aada. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.409077 10322 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.409225 10188 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.409397 9664 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.409797 12283 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.410270 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.411044 10059 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.411870 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.415568 12283 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.420125 9797 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.424667 12283 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.434067 12283 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.440954 9664 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.447381 9797 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.448166 9393 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.448559 10188 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.456434 9927 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.457299 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.474849 10321 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.475482 10059 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.483680 9533 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:40.487571 9927 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.488852 10059 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.489362 10188 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.489415 12283 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.489907 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.491799 9797 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.491815 10321 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.492332 12283 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.493410 9664 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.494055 12283 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.494344 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.494593 9533 consensus_queue.cc:237] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:40.497121 11712 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
W20240503 08:14:40.498099 9468 consensus_peers.cc:485] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fe97b80e1294c4791560daf991bfcdb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.497814 10321 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.500082 9797 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.502600 9929 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.503662 9393 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.504900 12490 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.505048 10191 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.505472 10059 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.506189 12474 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.508016 12283 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.510424 9929 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.510955 12474 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.511818 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3fe97b80e1294c4791560daf991bfcdb with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:40.513437 10191 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.514022 10059 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.515326 10321 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.515846 9929 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.516570 12283 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.519482 12481 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.520471 9666 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.520174 9927 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.520959 12481 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.524222 9797 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.524689 12490 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.528430 10191 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.529608 9392 catalog_manager.cc:5484] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.532385 10059 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.534962 9664 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:40.540786 9666 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.531229 10321 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.543481 9533 consensus_queue.cc:237] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:40.554090 9929 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.554860 10191 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.555222 10059 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.555428 12283 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.555721 12481 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.556514 9664 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.556747 10321 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.557233 11712 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.557623 12489 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.557811 11712 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.560475 12481 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:40.561014 10191 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:40.562001 9666 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:40.564529 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9428100e4a364805863419b22ca3c682 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:40.564999 10059 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:40.565299 9929 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:40.565300 10060 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:40.566280 10321 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:40.571444 9385 catalog_manager.cc:5484] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:40.578383 9666 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.579197 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.579942 10188 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.579964 9798 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.580549 12459 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.580797 9929 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.581436 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.581537 10321 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.581933 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.587870 12459 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:40.588354 9470 consensus_peers.cc:485] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9428100e4a364805863419b22ca3c682. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.590605 9533 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:40.591617 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:40.594106 9796 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:40.601450 9797 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.602211 12481 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.603693 9797 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:40.610559 12459 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.611565 9664 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.616930 9389 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.617429 9798 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.615760 9533 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.629222 9929 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.642707 10059 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:40.642724 10322 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.642802 10188 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:40.647104 10188 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:14:40.647280 9993 consensus_peers.cc:485] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43f3607501a84050962dc116210c09cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:40.647728 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.649638 10322 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.651118 9533 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.651142 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.651329 9798 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.651875 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.652112 12466 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.655340 9664 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.655917 12319 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.655616 12458 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.657600 9929 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:40.658104 12459 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:40.657990 9664 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.659554 9927 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.661257 9533 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.661352 10321 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.663802 9380 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 43f3607501a84050962dc116210c09cd with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:40.664799 10188 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.667193 9392 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.670204 9798 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:40.696331 12520 ts_tablet_manager.cc:902] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:40.698129 12520 tablet_copy_client.cc:308] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:41805
I20240503 08:14:40.707993 9818 tablet_copy_service.cc:140] P 8de08e679a904ee48ba659861c730725: Received BeginTabletCopySession request for tablet f13c7cfe35b3403f87940a124e6d9cda from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:35377)
I20240503 08:14:40.708231 9818 tablet_copy_service.cc:161] P 8de08e679a904ee48ba659861c730725: Beginning new tablet copy session on tablet f13c7cfe35b3403f87940a124e6d9cda from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:35377: session id = ca5fa388d5b84a0abac1c05275155dcf-f13c7cfe35b3403f87940a124e6d9cda
I20240503 08:14:40.711287 9818 tablet_copy_source_session.cc:205] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.717005 12520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c7cfe35b3403f87940a124e6d9cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.727977 12533 ts_tablet_manager.cc:902] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:40.729199 12520 tablet_copy_client.cc:783] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:40.729660 12520 tablet_copy_client.cc:647] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.733016 12520 tablet_copy_client.cc:523] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:40.737555 12520 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:40.754562 12520 log.cc:824] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:40.785777 12533 tablet_copy_client.cc:308] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.6:37175
I20240503 08:14:40.794512 10211 tablet_copy_service.cc:140] P fa60e870de844325bad4b9185eaee468: Received BeginTabletCopySession request for tablet cc9f0afdc01d4d03917b9040c2053426 from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:48377)
I20240503 08:14:40.794751 10211 tablet_copy_service.cc:161] P fa60e870de844325bad4b9185eaee468: Beginning new tablet copy session on tablet cc9f0afdc01d4d03917b9040c2053426 from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:48377: session id = 3a70e21b51a142bd8f7dc93fa58742fb-cc9f0afdc01d4d03917b9040c2053426
I20240503 08:14:40.799705 10211 tablet_copy_source_session.cc:205] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.804780 12533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f0afdc01d4d03917b9040c2053426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.822794 12533 tablet_copy_client.cc:783] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:40.823720 12533 tablet_copy_client.cc:647] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.827111 12533 tablet_copy_client.cc:523] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:40.831745 12533 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:40.839897 12520 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:40.840560 12520 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:40.841027 12520 ts_tablet_manager.cc:1367] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.104s user 0.064s sys 0.013s
I20240503 08:14:40.846838 12520 raft_consensus.cc:348] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:40.847404 12520 raft_consensus.cc:729] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:40.848001 12520 consensus_queue.cc:260] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:40.871841 12520 ts_tablet_manager.cc:1398] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.031s user 0.014s sys 0.016s
I20240503 08:14:40.873965 9818 tablet_copy_service.cc:342] P 8de08e679a904ee48ba659861c730725: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-f13c7cfe35b3403f87940a124e6d9cda received from {username='slave'} at 127.24.123.8:35377
I20240503 08:14:40.874145 9818 tablet_copy_service.cc:433] P 8de08e679a904ee48ba659861c730725: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-f13c7cfe35b3403f87940a124e6d9cda on tablet f13c7cfe35b3403f87940a124e6d9cda with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:40.875000 12533 log.cc:824] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:40.877837 12520 ts_tablet_manager.cc:902] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:40.888199 12559 ts_tablet_manager.cc:902] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:40.906647 12559 tablet_copy_client.cc:308] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43643
I20240503 08:14:40.907593 12520 tablet_copy_client.cc:308] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:40.914832 10080 tablet_copy_service.cc:140] P a1402a75453d4ccf992606dcce77dc51: Received BeginTabletCopySession request for tablet 1f3941801988415e9fc2620c5ae5aada from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:58265)
I20240503 08:14:40.915084 10080 tablet_copy_service.cc:161] P a1402a75453d4ccf992606dcce77dc51: Beginning new tablet copy session on tablet 1f3941801988415e9fc2620c5ae5aada from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:58265: session id = 3a70e21b51a142bd8f7dc93fa58742fb-1f3941801988415e9fc2620c5ae5aada
I20240503 08:14:40.917598 10080 tablet_copy_source_session.cc:205] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.918838 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 3fe97b80e1294c4791560daf991bfcdb from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:50757)
I20240503 08:14:40.919168 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 3fe97b80e1294c4791560daf991bfcdb from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:50757: session id = ca5fa388d5b84a0abac1c05275155dcf-3fe97b80e1294c4791560daf991bfcdb
I20240503 08:14:40.920331 12559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f3941801988415e9fc2620c5ae5aada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.922857 9555 tablet_copy_source_session.cc:205] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.925115 12559 tablet_copy_client.cc:783] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:40.925349 12559 tablet_copy_client.cc:647] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.925758 12520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.927686 12559 tablet_copy_client.cc:523] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:40.931008 12559 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:40.932255 12520 tablet_copy_client.cc:783] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:40.932556 12520 tablet_copy_client.cc:647] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:40.935180 12520 tablet_copy_client.cc:523] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:40.938812 12520 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:40.949520 12533 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:40.950121 12533 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:40.950627 12533 ts_tablet_manager.cc:1367] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.119s user 0.058s sys 0.021s
I20240503 08:14:40.955206 12569 ts_tablet_manager.cc:902] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:40.958667 12533 raft_consensus.cc:348] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:40.959411 12533 raft_consensus.cc:729] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:40.960116 12533 consensus_queue.cc:260] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:40.968766 12533 ts_tablet_manager.cc:1398] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20240503 08:14:40.969753 10211 tablet_copy_service.cc:342] P fa60e870de844325bad4b9185eaee468: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-cc9f0afdc01d4d03917b9040c2053426 received from {username='slave'} at 127.24.123.9:48377
I20240503 08:14:40.969936 10211 tablet_copy_service.cc:433] P fa60e870de844325bad4b9185eaee468: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-cc9f0afdc01d4d03917b9040c2053426 on tablet cc9f0afdc01d4d03917b9040c2053426 with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:40.977663 12569 tablet_copy_client.cc:308] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:40.988951 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 9428100e4a364805863419b22ca3c682 from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:59753)
I20240503 08:14:40.989215 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 9428100e4a364805863419b22ca3c682 from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:59753: session id = 925c339b047b4cf5bae840c5a357d3de-9428100e4a364805863419b22ca3c682
I20240503 08:14:40.992170 9555 tablet_copy_source_session.cc:205] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:40.996479 12559 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:40.996906 12559 tablet_bootstrap.cc:492] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:40.997172 12559 ts_tablet_manager.cc:1367] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.066s user 0.038s sys 0.002s
I20240503 08:14:40.997990 12569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:40.998099 12559 raft_consensus.cc:348] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:40.998432 12559 raft_consensus.cc:729] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:40.998652 12559 consensus_queue.cc:260] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:40.999269 12559 ts_tablet_manager.cc:1398] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:14:41.000041 10080 tablet_copy_service.cc:342] P a1402a75453d4ccf992606dcce77dc51: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-1f3941801988415e9fc2620c5ae5aada received from {username='slave'} at 127.24.123.9:58265
I20240503 08:14:41.000232 10080 tablet_copy_service.cc:433] P a1402a75453d4ccf992606dcce77dc51: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-1f3941801988415e9fc2620c5ae5aada on tablet 1f3941801988415e9fc2620c5ae5aada with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:41.012768 12569 tablet_copy_client.cc:783] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:41.013595 12569 tablet_copy_client.cc:647] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:41.014897 12520 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:41.015347 12520 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:41.015646 12520 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.077s user 0.053s sys 0.014s
I20240503 08:14:41.016433 12569 tablet_copy_client.cc:523] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:41.016607 12520 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:41.016969 12520 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:41.017212 12520 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:41.017937 12520 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:41.018738 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-3fe97b80e1294c4791560daf991bfcdb received from {username='slave'} at 127.24.123.8:50757
I20240503 08:14:41.018895 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-3fe97b80e1294c4791560daf991bfcdb on tablet 3fe97b80e1294c4791560daf991bfcdb with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:41.020572 12569 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:41.031237 12569 log.cc:824] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:41.063921 12569 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:41.064472 12569 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:41.065374 12569 ts_tablet_manager.cc:1367] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.045s user 0.046s sys 0.000s
I20240503 08:14:41.070318 12569 raft_consensus.cc:348] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:41.070793 12569 raft_consensus.cc:729] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:41.071429 12569 consensus_queue.cc:260] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:41.075004 12569 ts_tablet_manager.cc:1398] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240503 08:14:41.076032 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-9428100e4a364805863419b22ca3c682 received from {username='slave'} at 127.24.123.10:59753
I20240503 08:14:41.076159 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-9428100e4a364805863419b22ca3c682 on tablet 9428100e4a364805863419b22ca3c682 with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:41.151535 12559 ts_tablet_manager.cc:902] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:41.152663 12559 tablet_copy_client.cc:308] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43643
I20240503 08:14:41.153232 10080 tablet_copy_service.cc:140] P a1402a75453d4ccf992606dcce77dc51: Received BeginTabletCopySession request for tablet 43f3607501a84050962dc116210c09cd from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:58265)
I20240503 08:14:41.153391 10080 tablet_copy_service.cc:161] P a1402a75453d4ccf992606dcce77dc51: Beginning new tablet copy session on tablet 43f3607501a84050962dc116210c09cd from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:58265: session id = 3a70e21b51a142bd8f7dc93fa58742fb-43f3607501a84050962dc116210c09cd
I20240503 08:14:41.155320 10080 tablet_copy_source_session.cc:205] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:41.156365 12559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:41.160578 12559 tablet_copy_client.cc:783] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:41.160773 12559 tablet_copy_client.cc:647] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:41.162451 12559 tablet_copy_client.cc:523] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:41.165172 12559 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:41.202312 12559 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:41.202190 12224 raft_consensus.cc:1203] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:41.202765 12559 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:41.203074 12559 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.038s user 0.037s sys 0.002s
I20240503 08:14:41.203855 12559 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:41.204165 12559 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:41.204385 12559 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:41.204977 12559 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:41.206099 10080 tablet_copy_service.cc:342] P a1402a75453d4ccf992606dcce77dc51: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-43f3607501a84050962dc116210c09cd received from {username='slave'} at 127.24.123.9:58265
I20240503 08:14:41.206233 10080 tablet_copy_service.cc:433] P a1402a75453d4ccf992606dcce77dc51: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-43f3607501a84050962dc116210c09cd on tablet 43f3607501a84050962dc116210c09cd with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:41.250965 11958 raft_consensus.cc:1203] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:41.349892 12224 raft_consensus.cc:1203] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:41.360237 11958 raft_consensus.cc:1203] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:41.515892 12090 raft_consensus.cc:1203] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:41.692386 11958 raft_consensus.cc:1203] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:41.758844 12458 raft_consensus.cc:1050] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:41.759873 12458 consensus_queue.cc:237] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:41.762712 9798 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.762768 9666 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.763376 11958 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.763494 12617 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.765004 12617 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.765133 9535 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.765290 12459 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.765738 10322 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.765794 12459 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.767532 12617 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.769589 9927 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.770088 12679 raft_consensus.cc:1050] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:41.770156 12617 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.770520 10188 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.771065 12617 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.771306 12679 consensus_queue.cc:237] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:41.773478 12617 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.778380 9666 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.779404 9798 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.778796 11958 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.778703 9535 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.784598 9389 catalog_manager.cc:5484] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.803396 9535 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.803136 10188 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.804216 12224 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.805169 12148 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.805708 12148 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.810151 10322 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.812526 9927 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.820060 9666 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.820235 10059 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.820837 10322 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.821352 12148 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.821851 12148 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.822311 12148 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.825891 9927 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.826262 10188 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.826385 12148 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.827958 12148 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.829128 12148 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.832072 9927 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.835613 9535 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.835239 9666 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.837344 12224 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.844030 9393 catalog_manager.cc:5484] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.835577 10060 consensus_queue.cc:237] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:41.844352 10188 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.850189 11958 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.850957 12691 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.851851 9796 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.853170 9666 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.853209 9927 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.853654 12691 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.853894 12682 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.853780 10321 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.854342 10322 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.854713 12459 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.857213 12691 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.858500 12610 raft_consensus.cc:1050] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:41.858318 12459 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.858469 10059 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.860370 12610 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:41.860580 10322 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.861752 9927 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.864286 9666 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.866087 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f3941801988415e9fc2620c5ae5aada with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:41.866989 11958 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.870074 9796 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.870802 9390 catalog_manager.cc:5484] T 1f3941801988415e9fc2620c5ae5aada P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.889016 9535 raft_consensus.cc:1261] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.889644 12459 consensus_queue.cc:1035] T 1f3941801988415e9fc2620c5ae5aada P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.891703 9535 raft_consensus.cc:2932] T 1f3941801988415e9fc2620c5ae5aada P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:41.897397 12481 raft_consensus.cc:1050] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:41.898574 12481 consensus_queue.cc:237] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:41.903596 9796 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.904198 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.904954 12224 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.906015 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.910115 9798 consensus_queue.cc:237] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:41.912503 12223 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.912611 9533 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.913885 9666 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.913914 10059 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.914005 12695 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.914623 12700 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.914979 12695 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.919200 12679 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.920591 12700 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.924607 9533 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.926059 9666 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.926651 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.927553 10060 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.930317 10191 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.931188 10322 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.931911 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.932132 9927 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:41.932422 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.932783 12679 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:41.934442 9927 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.944697 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f13c7cfe35b3403f87940a124e6d9cda with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:41.946158 12223 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.947461 9393 catalog_manager.cc:5484] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.950249 9666 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.960285 12090 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.960978 11712 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.961522 10060 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.962471 9927 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.963297 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.963905 11712 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.973896 12328 raft_consensus.cc:1050] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:41.975167 12328 consensus_queue.cc:237] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:41.975824 10169 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f3941801988415e9fc2620c5ae5aada with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:41.976763 12733 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:41.978982 10321 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.979698 12731 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.981923 9927 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.982987 10189 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:41.986899 12721 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.987999 12721 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:41.996944 10171 tablet_service.cc:1514] Processing DeleteTablet for tablet f13c7cfe35b3403f87940a124e6d9cda with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:42.003698 11958 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.003727 10320 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.004248 12727 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.004448 9798 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.004951 9929 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.005156 10060 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.005242 9666 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.005661 12725 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.005832 9798 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.006534 12711 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.007054 12711 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.007424 12450 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.007918 9535 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.007920 12450 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.008671 12711 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.008987 12741 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.009359 10319 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.017596 10191 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.019656 10322 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.027155 9393 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.031620 12224 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.033272 12734 tablet_replica.cc:330] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:42.035193 12734 raft_consensus.cc:2229] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:42.035585 12734 raft_consensus.cc:2258] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:42.036788 12716 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.041270 12734 ts_tablet_manager.cc:1875] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:42.042966 12726 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.042905 9796 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.044266 9929 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.044703 9666 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
W20240503 08:14:42.049867 9470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43643, user_credentials={real_user=slave}} blocked reactor thread for 86713.6us
I20240503 08:14:42.050891 9387 catalog_manager.cc:5484] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.051190 11712 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.051275 11958 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.059301 10321 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.060348 10320 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.058462 9387 catalog_manager.cc:5484] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:42.060864 9927 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.063390 10059 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.064478 9666 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.066054 9796 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.070772 10191 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.074458 9929 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.077742 9535 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.075606 9798 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.089584 12734 ts_tablet_manager.cc:1888] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:42.090121 12734 log.cc:1196] T 1f3941801988415e9fc2620c5ae5aada P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/1f3941801988415e9fc2620c5ae5aada
I20240503 08:14:42.091203 12734 tablet_replica.cc:330] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:42.091517 12740 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.091609 10322 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.091781 9666 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.092269 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet 1f3941801988415e9fc2620c5ae5aada (table rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]) successfully deleted
I20240503 08:14:42.092792 12734 raft_consensus.cc:2229] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:42.093298 12734 raft_consensus.cc:2258] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:42.092727 12733 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.093724 12090 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.094641 12734 ts_tablet_manager.cc:1875] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:42.106159 12734 ts_tablet_manager.cc:1888] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:42.106388 12734 log.cc:1196] T f13c7cfe35b3403f87940a124e6d9cda P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/f13c7cfe35b3403f87940a124e6d9cda
I20240503 08:14:42.107165 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet f13c7cfe35b3403f87940a124e6d9cda (table rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]) successfully deleted
I20240503 08:14:42.121671 10058 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.126896 10191 consensus_queue.cc:237] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:42.128935 10059 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.131371 9798 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.132059 11958 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.133487 9666 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.134101 12727 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.133798 9664 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.134549 12727 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.134984 9535 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.134981 12727 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.137856 12725 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.122772 10060 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.143004 9798 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.144814 10321 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.145835 12693 raft_consensus.cc:1050] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:42.147068 12328 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.146993 12693 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:42.150027 12727 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.151104 10321 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.154789 9535 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.156725 9387 catalog_manager.cc:5484] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.160598 9379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc9f0afdc01d4d03917b9040c2053426 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:42.161191 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9f0afdc01d4d03917b9040c2053426 with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:42.162679 12763 tablet_replica.cc:330] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:42.163297 12763 raft_consensus.cc:2229] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:42.163822 12763 raft_consensus.cc:2258] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:42.163421 11958 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.165769 12763 ts_tablet_manager.cc:1875] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:42.167251 9664 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.172570 12763 ts_tablet_manager.cc:1888] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:42.172791 12763 log.cc:1196] T cc9f0afdc01d4d03917b9040c2053426 P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/cc9f0afdc01d4d03917b9040c2053426
I20240503 08:14:42.174226 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet cc9f0afdc01d4d03917b9040c2053426 (table rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]) successfully deleted
I20240503 08:14:42.176122 11958 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.176558 9798 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.177093 12459 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.177784 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.179003 9664 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.179553 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.180900 9535 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.181797 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.183849 10321 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.184536 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.186436 10058 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.189385 12328 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.191080 10058 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.199826 12691 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.205631 9665 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.207492 10191 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.207791 9929 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:42.208043 12693 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.208525 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:42.210546 9929 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.212049 10191 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.216835 9535 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.216964 10321 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.220986 11958 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.221544 9388 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.226477 9798 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.501436 10321 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:42.505106 10191 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.505281 9798 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.505782 9665 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.505748 12798 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.506325 9929 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.506983 12713 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.507503 10060 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.507555 12713 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.507798 12633 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.508482 12713 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.515673 9535 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.516947 12799 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.519773 12713 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.520361 9535 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.525455 9798 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.525858 10191 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.526922 9390 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:42.524893 10060 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.534468 9665 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.538352 9929 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.540997 10321 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:42.546516 9535 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.547195 12713 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.549311 9665 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.550153 12713 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.555444 9798 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.555845 10191 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.556411 12799 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.556715 12633 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.557128 9929 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.557648 12633 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.561574 9798 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.561853 10188 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.562495 10189 consensus_queue.cc:237] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:42.560846 12798 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.568068 9393 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.568645 9927 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.572717 10060 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.573426 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.575079 10060 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.578362 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e11a6197ccdd4e8d9a203efae4a94988 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:42.579257 9535 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
W20240503 08:14:42.583691 10257 consensus_peers.cc:485] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e11a6197ccdd4e8d9a203efae4a94988. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:42.585430 9665 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.598218 9535 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:42.600888 10188 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.601110 12224 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.601115 9798 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.601442 12481 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.601691 10321 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.601862 10322 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.601881 9665 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.601948 12481 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.602208 12740 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.602458 12733 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.603003 12733 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.602996 12328 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.606149 12733 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.607432 12224 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.607465 10322 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.610352 9929 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.611122 12727 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.611608 9664 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.612463 9798 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.613396 12725 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.613196 10189 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.613749 10058 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.615058 12727 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.616084 10060 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.616228 12740 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.618163 12328 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.618453 9387 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:42.619174 9535 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.619845 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fe97b80e1294c4791560daf991bfcdb with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:42.620095 9665 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.620451 9796 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.618342 10060 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:42.627136 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe97b80e1294c4791560daf991bfcdb with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:42.627548 12763 tablet_replica.cc:330] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:42.627753 12763 raft_consensus.cc:2229] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:42.628108 12763 raft_consensus.cc:2258] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:42.629262 12328 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.631055 12763 ts_tablet_manager.cc:1875] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:42.649447 12725 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.650684 12763 ts_tablet_manager.cc:1888] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:42.650895 12763 log.cc:1196] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/3fe97b80e1294c4791560daf991bfcdb
I20240503 08:14:42.650681 9665 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.651871 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet 3fe97b80e1294c4791560daf991bfcdb (table rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]) successfully deleted
I20240503 08:14:42.654594 9929 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.656605 10321 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.658108 9387 catalog_manager.cc:5484] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:42.663223 9535 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.666136 9798 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.668287 10058 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
W20240503 08:14:42.677732 12725 consensus_queue.cc:1159] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8de08e679a904ee48ba659861c730725" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20240503 08:14:42.678185 10189 consensus_queue.cc:237] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:42.682677 9535 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.683948 12706 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.684263 9929 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.684791 9798 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.684785 12725 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.685279 12706 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.686197 9664 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.687098 10058 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.688122 10321 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.689030 12725 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240503 08:14:42.689365 12706 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.690088 12328 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.693922 12328 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.694504 10321 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.696838 9666 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.700212 9535 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.700878 9379 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a61f59d188824bf3b610771b2fc05f06 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 08:14:42.704103 10126 consensus_peers.cc:485] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a61f59d188824bf3b610771b2fc05f06. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:42.704659 9798 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.705803 9385 catalog_manager.cc:5484] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.705900 9929 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.706441 9533 consensus_queue.cc:237] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:42.709389 9796 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.710053 12090 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.710585 10059 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.711475 12741 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.711613 9927 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.712035 12741 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.712625 12481 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.719468 10321 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.719812 10059 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.722083 12817 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.724740 12481 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.725576 12733 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.727171 10322 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.729985 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9428100e4a364805863419b22ca3c682 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:42.731057 9796 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.736912 9393 catalog_manager.cc:5484] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.742385 9665 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.742813 10169 tablet_service.cc:1514] Processing DeleteTablet for tablet 9428100e4a364805863419b22ca3c682 with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:42.743522 9927 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.746771 10059 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.747547 12090 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.753526 12733 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.755496 9665 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:42.761701 10058 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:42.764590 11958 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.764938 9798 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.765233 9665 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.765738 12693 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.766131 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.766834 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.770465 12831 tablet_replica.cc:330] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:42.770579 10189 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.770668 12831 raft_consensus.cc:2229] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:42.771660 12834 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.772078 12831 raft_consensus.cc:2258] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:42.773512 10321 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.774027 9533 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:42.776178 12682 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.778462 9533 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:42.780030 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43f3607501a84050962dc116210c09cd with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:42.780715 9665 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.781126 10189 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.785655 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.786167 12458 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:42.786010 11958 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.787539 9387 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.792068 12831 ts_tablet_manager.cc:1875] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:42.793329 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f3607501a84050962dc116210c09cd with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:42.793733 12763 tablet_replica.cc:330] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:42.793944 12763 raft_consensus.cc:2229] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:42.794190 12763 raft_consensus.cc:2258] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:42.795101 12763 ts_tablet_manager.cc:1875] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:42.800285 10321 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.801034 12763 ts_tablet_manager.cc:1888] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:42.801143 12763 log.cc:1196] T 43f3607501a84050962dc116210c09cd P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/43f3607501a84050962dc116210c09cd
I20240503 08:14:42.801793 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet 43f3607501a84050962dc116210c09cd (table rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]) successfully deleted
I20240503 08:14:42.805222 9798 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.805449 9535 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:42.807636 12831 ts_tablet_manager.cc:1888] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:42.807775 12831 log.cc:1196] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/9428100e4a364805863419b22ca3c682
I20240503 08:14:42.808827 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet 9428100e4a364805863419b22ca3c682 (table rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]) successfully deleted
I20240503 08:14:42.810861 9927 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.811043 10189 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.811525 12828 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.811772 12817 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.812913 9796 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.813548 12481 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.813797 10321 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.814302 9666 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.814774 10058 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:42.814765 12481 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.816527 12828 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.818255 12741 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.843494 12481 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.847412 10059 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.849779 10189 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.851598 9387 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:42.853199 10321 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.856305 9796 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.859606 9666 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.864045 9533 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:42.868309 10321 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.868371 10059 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:14:42.868671 9470 consensus_peers.cc:485] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1be499fcc2bf4716b16a5983950d3ed8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:42.869019 12740 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.869493 12740 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.870564 9665 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.871647 10189 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.871682 9796 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.871915 12741 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.872290 12740 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.872651 12741 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:42.876156 12481 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.876858 10321 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.879807 9929 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:42.880180 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1be499fcc2bf4716b16a5983950d3ed8 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:42.879837 9665 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.880928 10189 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.881095 9798 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.882594 12828 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:42.884088 9929 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:42.885159 9387 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:42.890244 9929 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:42.893376 10059 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:43.144349 12875 ts_tablet_manager.cc:902] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:43.145711 12875 tablet_copy_client.cc:308] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.7:42473
I20240503 08:14:43.151840 10342 tablet_copy_service.cc:140] P 0cb18b8b10f944e39829faf16e061fd6: Received BeginTabletCopySession request for tablet e11a6197ccdd4e8d9a203efae4a94988 from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:38815)
I20240503 08:14:43.152024 10342 tablet_copy_service.cc:161] P 0cb18b8b10f944e39829faf16e061fd6: Beginning new tablet copy session on tablet e11a6197ccdd4e8d9a203efae4a94988 from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:38815: session id = 925c339b047b4cf5bae840c5a357d3de-e11a6197ccdd4e8d9a203efae4a94988
I20240503 08:14:43.154438 10342 tablet_copy_source_session.cc:205] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:43.156412 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:43.161442 12875 tablet_copy_client.cc:783] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:43.161718 12875 tablet_copy_client.cc:647] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:43.163923 12875 tablet_copy_client.cc:523] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:43.167251 12875 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:43.183960 9665 consensus_queue.cc:237] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:43.192534 9798 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.193197 9533 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.194097 12816 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.194623 12816 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.212889 12875 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:43.214181 12875 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:43.214494 12875 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.047s user 0.040s sys 0.004s
I20240503 08:14:43.215482 12875 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:43.215749 12875 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:43.215950 12875 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:43.216430 12875 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:43.217239 10342 tablet_copy_service.cc:342] P 0cb18b8b10f944e39829faf16e061fd6: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-e11a6197ccdd4e8d9a203efae4a94988 received from {username='slave'} at 127.24.123.10:38815
I20240503 08:14:43.217423 10342 tablet_copy_service.cc:433] P 0cb18b8b10f944e39829faf16e061fd6: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-e11a6197ccdd4e8d9a203efae4a94988 on tablet e11a6197ccdd4e8d9a203efae4a94988 with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:43.219025 10059 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.219475 10189 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.219826 10321 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.220017 12889 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.220417 12816 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.220666 12889 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.221064 9929 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.221737 12816 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.242358 12816 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.248682 9798 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.252679 9927 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.255939 10189 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.257534 9387 catalog_manager.cc:5484] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.259577 10059 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.268352 9533 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.277153 9665 consensus_queue.cc:237] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:43.280289 10189 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.280541 9927 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.280835 12900 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.281198 12816 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.285863 10321 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.288103 9798 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.288254 9533 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.288655 12816 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.290261 12875 ts_tablet_manager.cc:902] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:43.291673 10322 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.296222 12816 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.297156 12816 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.289216 10059 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.298408 12816 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:43.299180 9798 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:43.305925 12900 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.307399 10059 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:43.307893 9388 catalog_manager.cc:5484] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:43.309517 9927 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:43.314119 10189 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:43.320569 10059 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:43.322553 9380 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8506211351284b7a8e2a166b8083977a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:43.323901 9533 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:43.327756 9927 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.328907 12836 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:14:43.329022 9602 consensus_peers.cc:485] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8506211351284b7a8e2a166b8083977a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.331455 10322 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.331533 9665 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.332115 12836 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.332824 9798 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.332823 12836 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.332930 10322 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:43.333781 12836 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.333935 9535 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.334646 12836 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.340037 10189 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.341068 12834 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.348338 12834 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:43.355408 9927 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:43.357326 10189 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:43.361340 9393 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:43.363078 9535 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:43.363112 9666 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:43.368763 12924 ts_tablet_manager.cc:902] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:43.384295 10058 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:43.385367 12875 tablet_copy_client.cc:308] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.6:37175
I20240503 08:14:43.388670 10189 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.391124 9666 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.389257 12912 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240503 08:14:43.394676 10211 tablet_copy_service.cc:140] P fa60e870de844325bad4b9185eaee468: Received BeginTabletCopySession request for tablet a61f59d188824bf3b610771b2fc05f06 from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:34373)
I20240503 08:14:43.394860 10211 tablet_copy_service.cc:161] P fa60e870de844325bad4b9185eaee468: Beginning new tablet copy session on tablet a61f59d188824bf3b610771b2fc05f06 from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:34373: session id = 925c339b047b4cf5bae840c5a357d3de-a61f59d188824bf3b610771b2fc05f06
I20240503 08:14:43.395509 12926 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.397854 10211 tablet_copy_source_session.cc:205] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:43.398623 9927 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.399415 12923 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.401075 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f59d188824bf3b610771b2fc05f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:43.411430 12875 tablet_copy_client.cc:783] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:43.411695 12875 tablet_copy_client.cc:647] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:43.412204 9535 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.413378 9798 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.413622 12875 tablet_copy_client.cc:523] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:43.413834 12926 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.417109 12875 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:43.418305 9380 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0f6758eb87b047d29a060fcbfb775903 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:43.420118 10189 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.424132 10319 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.423616 9927 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.425869 12923 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.426762 10319 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:43.430589 12924 tablet_copy_client.cc:308] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:43.427429 9665 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.431469 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 1be499fcc2bf4716b16a5983950d3ed8 from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:59753)
I20240503 08:14:43.431627 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 1be499fcc2bf4716b16a5983950d3ed8 from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:59753: session id = 925c339b047b4cf5bae840c5a357d3de-1be499fcc2bf4716b16a5983950d3ed8
I20240503 08:14:43.434587 9555 tablet_copy_source_session.cc:205] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:43.434561 12927 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.435081 12927 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.434854 9388 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.436748 10319 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.437062 9533 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.437391 9796 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:43.441221 9796 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.442327 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:14:43.446283 9993 consensus_peers.cc:485] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f6758eb87b047d29a060fcbfb775903. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.449855 12924 tablet_copy_client.cc:783] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:43.450109 12924 tablet_copy_client.cc:647] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:43.455157 12924 tablet_copy_client.cc:523] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:43.459645 12924 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:43.499475 9796 consensus_queue.cc:237] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:43.501587 12875 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:43.502086 12875 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:43.502286 9533 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.502477 12875 ts_tablet_manager.cc:1367] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.085s user 0.053s sys 0.003s
I20240503 08:14:43.503391 12875 raft_consensus.cc:348] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:43.503708 12875 raft_consensus.cc:729] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:43.503799 10319 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.503891 9666 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.503930 12875 consensus_queue.cc:260] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:43.504194 12700 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.504599 12875 ts_tablet_manager.cc:1398] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:43.505370 10211 tablet_copy_service.cc:342] P fa60e870de844325bad4b9185eaee468: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-a61f59d188824bf3b610771b2fc05f06 received from {username='slave'} at 127.24.123.10:34373
I20240503 08:14:43.505497 10211 tablet_copy_service.cc:433] P fa60e870de844325bad4b9185eaee468: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-a61f59d188824bf3b610771b2fc05f06 on tablet a61f59d188824bf3b610771b2fc05f06 with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:43.505540 12726 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.506336 12700 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.509898 10058 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.510798 12726 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.511003 10189 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.511518 12726 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.513636 9927 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:43.514312 12700 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:43.545357 12924 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:43.545841 12924 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:43.545471 12700 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.546151 12924 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.087s user 0.046s sys 0.014s
I20240503 08:14:43.547024 12924 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:43.547430 12924 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:43.546993 9665 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.547639 12924 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:43.548341 12924 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:43.549005 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-1be499fcc2bf4716b16a5983950d3ed8 received from {username='slave'} at 127.24.123.10:59753
I20240503 08:14:43.549155 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-1be499fcc2bf4716b16a5983950d3ed8 on tablet 1be499fcc2bf4716b16a5983950d3ed8 with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:43.551787 10059 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.551909 9533 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.556046 9388 catalog_manager.cc:5484] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.562073 10189 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.569841 9927 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.581918 10319 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:43.583369 9796 consensus_queue.cc:237] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:43.586916 9927 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.587527 12726 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:14:43.588528 9731 consensus_peers.cc:485] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34be73cc40264224ad798d7518c45ed1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:43.588770 10189 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.589298 12949 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.589395 10319 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.589841 10059 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.589908 9666 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.590204 12948 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.590401 9533 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:43.590802 12948 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.592001 12726 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.592262 12944 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:43.595566 12949 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.596452 10059 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.596698 10319 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.596850 9666 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.597994 9533 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.603049 9388 catalog_manager.cc:5484] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.608458 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 34be73cc40264224ad798d7518c45ed1 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:43.610080 10189 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.613713 9927 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:43.645516 12090 raft_consensus.cc:1203] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:43.768559 12090 raft_consensus.cc:1203] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:43.771258 12986 ts_tablet_manager.cc:902] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:43.773602 12986 tablet_copy_client.cc:308] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43643
I20240503 08:14:43.779906 10080 tablet_copy_service.cc:140] P a1402a75453d4ccf992606dcce77dc51: Received BeginTabletCopySession request for tablet 0f6758eb87b047d29a060fcbfb775903 from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:41803)
I20240503 08:14:43.780079 10080 tablet_copy_service.cc:161] P a1402a75453d4ccf992606dcce77dc51: Beginning new tablet copy session on tablet 0f6758eb87b047d29a060fcbfb775903 from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:41803: session id = ca5fa388d5b84a0abac1c05275155dcf-0f6758eb87b047d29a060fcbfb775903
I20240503 08:14:43.782096 10080 tablet_copy_source_session.cc:205] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:43.783406 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:43.787763 12986 tablet_copy_client.cc:783] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:43.787997 12986 tablet_copy_client.cc:647] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:43.789654 12986 tablet_copy_client.cc:523] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:43.792145 12986 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:43.822623 12986 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:43.823037 12986 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:43.823308 12986 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.010s
I20240503 08:14:43.824107 12986 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:43.824410 12986 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:43.824623 12986 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:43.825455 12986 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:14:43.826125 10080 tablet_copy_service.cc:342] P a1402a75453d4ccf992606dcce77dc51: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-0f6758eb87b047d29a060fcbfb775903 received from {username='slave'} at 127.24.123.8:41803
I20240503 08:14:43.826274 10080 tablet_copy_service.cc:433] P a1402a75453d4ccf992606dcce77dc51: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-0f6758eb87b047d29a060fcbfb775903 on tablet 0f6758eb87b047d29a060fcbfb775903 with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:43.830360 12090 raft_consensus.cc:1203] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:43.879101 12924 ts_tablet_manager.cc:902] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123)
I20240503 08:14:43.880193 12924 tablet_copy_client.cc:308] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:36123
I20240503 08:14:43.886054 9687 tablet_copy_service.cc:140] P d1eb5e69c6264a20a6e8297cc5deb8d1: Received BeginTabletCopySession request for tablet 8506211351284b7a8e2a166b8083977a from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:34183)
I20240503 08:14:43.886284 9687 tablet_copy_service.cc:161] P d1eb5e69c6264a20a6e8297cc5deb8d1: Beginning new tablet copy session on tablet 8506211351284b7a8e2a166b8083977a from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:34183: session id = 925c339b047b4cf5bae840c5a357d3de-8506211351284b7a8e2a166b8083977a
I20240503 08:14:43.888907 9687 tablet_copy_source_session.cc:205] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:43.890594 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8506211351284b7a8e2a166b8083977a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:43.894775 12924 tablet_copy_client.cc:783] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:43.895037 12924 tablet_copy_client.cc:647] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:43.896929 12924 tablet_copy_client.cc:523] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:43.899595 12924 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:43.942854 12924 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:43.943361 12924 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:43.943648 12924 ts_tablet_manager.cc:1367] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.044s user 0.030s sys 0.012s
I20240503 08:14:43.944504 12924 raft_consensus.cc:348] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:43.944787 12924 raft_consensus.cc:729] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:43.945024 12924 consensus_queue.cc:260] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:43.945645 12924 ts_tablet_manager.cc:1398] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:43.946465 9687 tablet_copy_service.cc:342] P d1eb5e69c6264a20a6e8297cc5deb8d1: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-8506211351284b7a8e2a166b8083977a received from {username='slave'} at 127.24.123.10:34183
I20240503 08:14:43.946650 9687 tablet_copy_service.cc:433] P d1eb5e69c6264a20a6e8297cc5deb8d1: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-8506211351284b7a8e2a166b8083977a on tablet 8506211351284b7a8e2a166b8083977a with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:44.066653 12986 ts_tablet_manager.cc:902] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:44.068621 12986 tablet_copy_client.cc:308] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:41805
I20240503 08:14:44.069322 9818 tablet_copy_service.cc:140] P 8de08e679a904ee48ba659861c730725: Received BeginTabletCopySession request for tablet 34be73cc40264224ad798d7518c45ed1 from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:35377)
I20240503 08:14:44.069480 9818 tablet_copy_service.cc:161] P 8de08e679a904ee48ba659861c730725: Beginning new tablet copy session on tablet 34be73cc40264224ad798d7518c45ed1 from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:35377: session id = ca5fa388d5b84a0abac1c05275155dcf-34be73cc40264224ad798d7518c45ed1
I20240503 08:14:44.071630 9818 tablet_copy_source_session.cc:205] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:44.072842 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be73cc40264224ad798d7518c45ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:44.078061 12986 tablet_copy_client.cc:783] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:44.078297 12986 tablet_copy_client.cc:647] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:44.079861 12986 tablet_copy_client.cc:523] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:44.082386 12986 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:44.113297 12986 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:44.113690 12986 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:44.113910 12986 ts_tablet_manager.cc:1367] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.003s
I20240503 08:14:44.114576 12986 raft_consensus.cc:348] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:44.114789 12986 raft_consensus.cc:729] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:44.115061 12986 consensus_queue.cc:260] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:44.115649 12986 ts_tablet_manager.cc:1398] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:14:44.116425 9818 tablet_copy_service.cc:342] P 8de08e679a904ee48ba659861c730725: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-34be73cc40264224ad798d7518c45ed1 received from {username='slave'} at 127.24.123.8:35377
I20240503 08:14:44.116541 9818 tablet_copy_service.cc:433] P 8de08e679a904ee48ba659861c730725: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-34be73cc40264224ad798d7518c45ed1 on tablet 34be73cc40264224ad798d7518c45ed1 with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:44.143065 12813 raft_consensus.cc:1050] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:44.144340 12813 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:44.147307 10059 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.147807 12090 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.148182 13037 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.148509 9666 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.148533 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.148813 10188 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.149087 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.149379 9796 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.149974 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.150483 12963 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.150645 9533 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.153264 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.153414 9927 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.154031 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.167007 12813 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.167665 12090 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.171257 9533 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.172505 10188 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.174410 10059 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.179235 9798 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.179548 9393 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:44.182355 9927 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.182008 9666 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.211653 10319 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:44.214856 10059 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.214845 12090 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.215670 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.216171 12963 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.217482 9796 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.218222 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.219342 9927 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.220532 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.221262 13037 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.223320 10059 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.223958 9927 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.230938 9391 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.231910 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e11a6197ccdd4e8d9a203efae4a94988 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:44.236256 10169 tablet_service.cc:1514] Processing DeleteTablet for tablet e11a6197ccdd4e8d9a203efae4a94988 with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:44.237869 9666 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.239004 9533 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.238972 13055 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.239789 12813 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.240530 13057 tablet_replica.cc:330] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:44.240749 13057 raft_consensus.cc:2229] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:44.241178 13057 raft_consensus.cc:2258] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:44.240819 9666 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.241250 12090 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.244169 13057 ts_tablet_manager.cc:1875] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:44.246294 9535 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.253993 13057 ts_tablet_manager.cc:1888] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:44.254225 13057 log.cc:1196] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/e11a6197ccdd4e8d9a203efae4a94988
I20240503 08:14:44.255745 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet e11a6197ccdd4e8d9a203efae4a94988 (table rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]) successfully deleted
I20240503 08:14:44.263666 9796 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.308065 12090 raft_consensus.cc:1203] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:44.314131 12224 raft_consensus.cc:1203] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:44.350181 12706 raft_consensus.cc:1050] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:44.351050 12706 consensus_queue.cc:237] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:44.353770 10319 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.354416 9927 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.354682 9796 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.354722 12994 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.355415 13084 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.355851 12994 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.356532 10059 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.356535 12090 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.357257 9666 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.358085 9535 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.358101 12706 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.358390 13083 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.359323 13083 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.361202 13083 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.387087 12741 raft_consensus.cc:1050] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:44.388300 12741 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:44.391625 12706 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.392319 9666 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.394690 10058 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.395318 10322 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.398332 9927 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.398934 12733 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.400067 12090 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.401489 12733 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.401598 9391 catalog_manager.cc:5484] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.410048 10188 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.410632 12733 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.414528 12090 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.415212 9929 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.419845 9796 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.411424 9535 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.413707 10319 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.422326 9798 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.423454 9666 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.425010 10188 consensus_queue.cc:237] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:44.426532 10059 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.429279 9798 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.429796 9666 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.429972 10319 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.430447 13083 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.431020 13083 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.431692 12090 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.431815 13085 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.432546 13085 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.436475 13085 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.438027 13096 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.440112 12733 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.440593 12733 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.442116 10322 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.445597 9379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a61f59d188824bf3b610771b2fc05f06 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:44.447832 12090 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.447897 9666 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.456689 9387 catalog_manager.cc:5484] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.461724 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet a61f59d188824bf3b610771b2fc05f06 with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:44.498764 10059 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.501008 13085 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:44.503010 10058 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
W20240503 08:14:44.513768 9471 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:38565, user_credentials={real_user=slave}} blocked reactor thread for 93615.6us
I20240503 08:14:44.515146 13098 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.519770 10188 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.526063 9387 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.527139 13105 tablet_replica.cc:330] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:44.527420 13105 raft_consensus.cc:2229] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:44.529574 13105 raft_consensus.cc:2258] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:14:44.530696 9470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.2:36123, user_credentials={real_user=slave}} blocked reactor thread for 106723us
I20240503 08:14:44.532754 12733 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.436913 9796 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.548367 13105 ts_tablet_manager.cc:1875] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:44.548314 9929 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.549887 10321 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.550087 12090 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.555063 9533 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:44.553640 9666 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.559098 13085 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20240503 08:14:44.562826 10059 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.563117 9533 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.565941 13105 ts_tablet_manager.cc:1888] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:44.566136 13105 log.cc:1196] T a61f59d188824bf3b610771b2fc05f06 P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/a61f59d188824bf3b610771b2fc05f06
I20240503 08:14:44.567024 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet a61f59d188824bf3b610771b2fc05f06 (table rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]) successfully deleted
I20240503 08:14:44.567236 9796 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.584074 12224 raft_consensus.cc:1203] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:44.601271 9929 consensus_queue.cc:237] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:44.605054 10058 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:44.605041 9796 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:44.605037 10321 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:44.605656 13117 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:44.606384 10189 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:44.606379 13117 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:44.607182 13117 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:44.607563 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:44.618598 9666 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:44.619257 9533 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:44.619238 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:44.619832 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:44.627521 12815 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:44.630470 10058 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:44.631326 9535 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:44.632984 10321 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:44.637142 9666 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:44.637288 9391 catalog_manager.cc:5484] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "06f1b17731834350a289de93abab8542" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:44.645599 9796 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:44.647320 10189 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:44.668852 9929 consensus_queue.cc:237] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:44.673164 9666 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:44.673933 10058 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:44.674211 10189 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:44.674845 13130 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:44.675773 13131 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:44.675942 10321 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:44.676390 9533 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:44.676640 9796 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:44.677727 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:44.678063 13118 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:44.678500 13131 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:44.678939 13118 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:44.680938 13117 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:44.684183 10189 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:44.686347 9533 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:44.686364 10321 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:44.686365 9796 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
W20240503 08:14:44.690608 9862 consensus_peers.cc:485] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd41cbc6c69441c1af5c36c382cfe366. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:44.692369 9387 catalog_manager.cc:5484] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "06f1b17731834350a289de93abab8542" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:44.693938 9666 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:44.695619 9381 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dd41cbc6c69441c1af5c36c382cfe366 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:44.695875 10058 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:44.848325 13066 raft_consensus.cc:1050] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:44.849401 13066 consensus_queue.cc:237] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:44.853049 12090 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.854197 9798 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.856550 13159 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.859023 13159 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.860733 10189 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.863652 10321 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.867447 13159 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20240503 08:14:44.870568 9929 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.871800 13165 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.875058 13163 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.877411 9533 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.878046 13165 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.884253 13159 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.887852 10058 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.888597 13166 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.888267 9533 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.891327 10189 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.893522 12090 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.895423 10058 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.895716 9390 catalog_manager.cc:5484] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:44.896436 9798 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.899654 9929 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.901713 12836 raft_consensus.cc:1050] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:44.902864 12836 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:44.903998 10321 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:44.912446 10322 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.912652 12224 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.913158 12927 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.913712 12927 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.919056 10189 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.919721 12836 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.920053 9666 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.920950 9798 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.921111 9533 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.921644 12836 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.921972 12927 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.923291 9929 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:44.923840 12927 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.927553 13176 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:44.956107 13176 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:44.961828 10189 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:44.962044 12224 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:44.965377 9929 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:44.968746 9391 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf reported cstate change: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:44.984680 9798 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:44.989929 10322 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:44.996807 9533 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:44.998567 9666 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.004913 10058 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:45.007601 9666 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.007663 12223 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.008139 13188 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.008663 13175 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.008812 9929 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.009291 13175 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.010859 9798 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.012296 9535 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.013386 12836 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.013938 12836 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.016695 13188 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.017463 9798 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.018024 9535 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.021042 9533 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:45.023442 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f6758eb87b047d29a060fcbfb775903 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:45.024641 9926 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.024742 12090 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.025257 13101 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.025621 9666 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.025715 10321 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.025626 13096 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.026400 12741 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.026973 12741 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.026607 9391 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.029951 10059 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.030364 9798 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.032887 13096 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.033077 10322 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.033598 13184 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.032780 9927 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.034390 13106 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.035240 13106 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.035379 10321 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.039346 12223 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.042199 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1be499fcc2bf4716b16a5983950d3ed8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:45.041767 12090 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.043648 10169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6758eb87b047d29a060fcbfb775903 with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:45.045243 13035 raft_consensus.cc:1050] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:45.045125 9666 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.045331 10322 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.046186 13035 consensus_queue.cc:237] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:45.049029 9665 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.051251 9798 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.051627 9929 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.055109 9389 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:45.061194 10171 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:45.061244 10060 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.067940 9535 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:45.068886 13111 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:45.071880 9929 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:45.072851 9666 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:45.073732 13035 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:45.074141 10058 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:45.075155 10188 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:45.076009 13111 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:45.076332 13035 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:45.076889 13035 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:45.080793 13191 tablet_replica.cc:330] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:45.081490 13191 raft_consensus.cc:2229] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:45.082119 13191 raft_consensus.cc:2258] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:45.082703 10322 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:45.083335 13035 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:45.084097 13191 ts_tablet_manager.cc:1875] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:45.084292 12223 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:45.086520 13035 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:45.091661 13191 ts_tablet_manager.cc:1888] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:45.097905 13191 log.cc:1196] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/0f6758eb87b047d29a060fcbfb775903
I20240503 08:14:45.097944 13035 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.099871 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet 0f6758eb87b047d29a060fcbfb775903 (table rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]) successfully deleted
I20240503 08:14:45.100544 13191 tablet_replica.cc:330] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:45.101119 13191 raft_consensus.cc:2229] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:45.101877 13191 raft_consensus.cc:2258] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:45.102962 10188 consensus_queue.cc:237] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:45.104097 13191 ts_tablet_manager.cc:1875] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:45.104167 10060 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.105619 10189 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.110951 9387 catalog_manager.cc:5484] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.112689 9535 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.115265 9929 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.121285 13191 ts_tablet_manager.cc:1888] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:45.122627 13191 log.cc:1196] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/1be499fcc2bf4716b16a5983950d3ed8
I20240503 08:14:45.123627 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet 1be499fcc2bf4716b16a5983950d3ed8 (table rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]) successfully deleted
I20240503 08:14:45.127252 10321 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.127272 9929 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.127815 13084 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.127924 9798 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.128491 9535 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.129083 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.129263 9665 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.129673 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.130125 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.130538 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.133128 10060 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.133668 13085 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.139032 12223 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.141372 10322 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.152086 9667 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.159652 9796 consensus_queue.cc:237] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:45.164400 10188 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.165000 13111 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.172199 9533 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.172209 12223 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.173332 13209 ts_tablet_manager.cc:902] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer 06f1b17731834350a289de93abab8542 (127.24.123.4:38565)
I20240503 08:14:45.173590 12949 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.174336 12949 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.174844 13085 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.177597 9535 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.185225 9392 catalog_manager.cc:5484] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.186697 10058 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.190022 10322 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.191314 10188 consensus_queue.cc:237] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:45.198777 9665 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.199617 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.200860 9665 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.201603 9535 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.202193 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:14:45.204470 10124 consensus_peers.cc:485] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 014f053652224d60ac8ef85064f0cabe. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:45.204888 10322 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.205041 10058 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.205528 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.208331 13084 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.209743 13085 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:45.210301 9665 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:45.216403 10322 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:45.217413 9387 catalog_manager.cc:5484] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:45.222782 12949 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.227454 9533 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.231420 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 34be73cc40264224ad798d7518c45ed1 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:45.234593 13209 tablet_copy_client.cc:308] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:38565
I20240503 08:14:45.235003 9389 catalog_manager.cc:5484] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.240998 9379 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 014f053652224d60ac8ef85064f0cabe with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:45.241278 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 34be73cc40264224ad798d7518c45ed1 with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:45.242501 9535 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:45.253922 9929 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.254693 13084 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.256693 10058 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:45.256306 9927 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.263309 13219 tablet_replica.cc:330] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:45.266620 13219 raft_consensus.cc:2229] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:45.267161 12224 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.267925 10322 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.269301 13219 raft_consensus.cc:2258] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:45.271764 13035 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.272569 9949 tablet_copy_service.cc:140] P 06f1b17731834350a289de93abab8542: Received BeginTabletCopySession request for tablet dd41cbc6c69441c1af5c36c382cfe366 from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:37747)
I20240503 08:14:45.272795 9949 tablet_copy_service.cc:161] P 06f1b17731834350a289de93abab8542: Beginning new tablet copy session on tablet dd41cbc6c69441c1af5c36c382cfe366 from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:37747: session id = 3a70e21b51a142bd8f7dc93fa58742fb-dd41cbc6c69441c1af5c36c382cfe366
I20240503 08:14:45.277480 10188 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.278542 9665 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.279886 10058 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.282593 10322 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.283164 9798 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.284086 13035 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.284823 13035 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.285281 13085 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.286707 9949 tablet_copy_source_session.cc:205] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:45.286549 10058 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.290215 13209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41cbc6c69441c1af5c36c382cfe366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:45.294183 9665 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:45.296928 13209 tablet_copy_client.cc:783] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:45.297230 13209 tablet_copy_client.cc:647] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:45.299336 13209 tablet_copy_client.cc:523] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:45.302762 13209 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:45.304783 9796 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.310351 13219 ts_tablet_manager.cc:1875] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:45.315898 9927 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:45.325893 13219 ts_tablet_manager.cc:1888] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:45.328405 9796 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:45.334980 13219 log.cc:1196] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/34be73cc40264224ad798d7518c45ed1
I20240503 08:14:45.336262 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet 34be73cc40264224ad798d7518c45ed1 (table rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]) successfully deleted
I20240503 08:14:45.360807 13209 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:45.361274 13209 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:45.361569 13209 ts_tablet_manager.cc:1367] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.059s user 0.037s sys 0.007s
I20240503 08:14:45.362249 13209 raft_consensus.cc:348] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:45.362543 13209 raft_consensus.cc:729] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:45.362741 13209 consensus_queue.cc:260] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:45.363551 13209 ts_tablet_manager.cc:1398] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:14:45.364500 9949 tablet_copy_service.cc:342] P 06f1b17731834350a289de93abab8542: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-dd41cbc6c69441c1af5c36c382cfe366 received from {username='slave'} at 127.24.123.9:37747
I20240503 08:14:45.364686 9949 tablet_copy_service.cc:433] P 06f1b17731834350a289de93abab8542: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-dd41cbc6c69441c1af5c36c382cfe366 on tablet dd41cbc6c69441c1af5c36c382cfe366 with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:45.517524 9665 consensus_queue.cc:237] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:45.520205 9535 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.520226 9798 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.520319 10322 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.520385 10188 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.521136 13255 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.521737 13256 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.521871 10058 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.521867 12090 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:45.522042 13255 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.522549 13255 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.522847 13167 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.523447 13167 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:45.526782 13254 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.527874 10058 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.528366 12090 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.531685 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8506211351284b7a8e2a166b8083977a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:45.532692 9798 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.533267 10322 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.537087 9392 catalog_manager.cc:5484] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.540014 10188 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.540261 9535 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:45.541661 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 8506211351284b7a8e2a166b8083977a with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:45.542520 13219 tablet_replica.cc:330] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:45.542711 13219 raft_consensus.cc:2229] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:45.543150 13219 raft_consensus.cc:2258] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:45.545495 13219 ts_tablet_manager.cc:1875] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:45.551548 13219 ts_tablet_manager.cc:1888] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:45.551699 13219 log.cc:1196] T 8506211351284b7a8e2a166b8083977a P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/8506211351284b7a8e2a166b8083977a
I20240503 08:14:45.552464 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet 8506211351284b7a8e2a166b8083977a (table rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]) successfully deleted
I20240503 08:14:45.623622 10322 consensus_queue.cc:237] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:45.627440 11958 raft_consensus.cc:1203] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:45.631490 9798 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.632247 9535 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.632432 9665 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.633420 9927 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.633426 10188 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.634382 10058 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.634506 13059 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.635085 13220 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.635660 13220 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.637220 13059 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.650389 13268 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.661646 13220 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.668761 13268 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:45.673487 9927 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:45.672523 9796 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:45.674273 9535 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:45.680475 9389 catalog_manager.cc:5484] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:45.682595 10188 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:45.682500 9665 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:45.694152 10322 consensus_queue.cc:237] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:45.697783 9927 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.698370 9665 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.697589 10058 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:45.699043 13059 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.699333 13269 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.699734 9535 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.700310 13059 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.704849 10060 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.705083 10188 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.705482 13268 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.705722 13059 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.710855 13059 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.711283 10188 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.716990 9796 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.717638 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fdb88bebfdab4e54abc5dba5834a3298 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:45.718384 13269 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.718113 9391 catalog_manager.cc:5484] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.718550 10060 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.719527 9927 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.720091 9796 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.726408 9667 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.718576 9535 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
W20240503 08:14:45.740358 10255 consensus_peers.cc:485] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdb88bebfdab4e54abc5dba5834a3298. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:45.753657 9796 consensus_queue.cc:237] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:45.756778 9927 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.756778 10322 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.757407 13208 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.757679 9667 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.757786 10060 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.757992 13208 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.758453 13111 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.758728 13207 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.761073 9535 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.761958 13111 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.763490 10188 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.764096 13111 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.775483 13209 ts_tablet_manager.cc:902] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:45.779861 13207 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.780928 10322 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.783447 9927 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.785147 10189 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.789415 9389 catalog_manager.cc:5484] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.791206 9535 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.792948 10058 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.799305 9665 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.804751 9798 consensus_queue.cc:237] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:45.807708 9535 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.808125 10189 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:14:45.808393 9731 consensus_peers.cc:485] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a2867608f0a4d69a3e169a59e358d10. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:45.808554 13035 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.808861 12949 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.814213 9665 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.814715 13035 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.815672 9929 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.817257 12949 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.819559 10058 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.819697 10322 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.819636 12949 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.820525 13296 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.821097 9927 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.822324 10322 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.823421 13208 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.824218 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a2867608f0a4d69a3e169a59e358d10 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:45.824733 9665 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.824843 10058 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.827087 9535 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.828027 10188 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.833240 9387 catalog_manager.cc:5484] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:45.854243 13209 tablet_copy_client.cc:308] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.6:37175
I20240503 08:14:45.855072 10211 tablet_copy_service.cc:140] P fa60e870de844325bad4b9185eaee468: Received BeginTabletCopySession request for tablet 014f053652224d60ac8ef85064f0cabe from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:48377)
I20240503 08:14:45.855244 10211 tablet_copy_service.cc:161] P fa60e870de844325bad4b9185eaee468: Beginning new tablet copy session on tablet 014f053652224d60ac8ef85064f0cabe from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:48377: session id = 3a70e21b51a142bd8f7dc93fa58742fb-014f053652224d60ac8ef85064f0cabe
I20240503 08:14:45.857611 10211 tablet_copy_source_session.cc:205] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:45.859310 13209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014f053652224d60ac8ef85064f0cabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:45.870903 13209 tablet_copy_client.cc:783] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:45.871161 13209 tablet_copy_client.cc:647] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:45.872917 13209 tablet_copy_client.cc:523] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:45.876497 13209 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:45.883973 9535 consensus_queue.cc:237] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:45.888365 9927 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.888484 10322 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.888484 10188 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.889757 12741 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.890385 12741 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.890992 12741 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.904347 10058 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.904397 9665 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.905006 9798 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:45.905040 13106 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.906558 13106 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.910131 13106 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:45.916221 12741 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.920753 10058 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.927145 10188 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.928618 9389 catalog_manager.cc:5484] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:45.930336 13209 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:45.931463 13209 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:45.931756 13209 ts_tablet_manager.cc:1367] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.055s user 0.040s sys 0.007s
I20240503 08:14:45.931838 9927 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.932564 13209 raft_consensus.cc:348] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:45.932897 13209 raft_consensus.cc:729] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:45.932286 9798 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.933131 13209 consensus_queue.cc:260] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:45.934345 13209 ts_tablet_manager.cc:1398] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:14:45.936198 10211 tablet_copy_service.cc:342] P fa60e870de844325bad4b9185eaee468: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-014f053652224d60ac8ef85064f0cabe received from {username='slave'} at 127.24.123.9:48377
I20240503 08:14:45.936345 10211 tablet_copy_service.cc:433] P fa60e870de844325bad4b9185eaee468: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-014f053652224d60ac8ef85064f0cabe on tablet 014f053652224d60ac8ef85064f0cabe with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:45.941490 10322 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.948163 9665 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:45.954005 9535 consensus_queue.cc:237] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
W20240503 08:14:45.958513 9468 consensus_peers.cc:485] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a9fcac8b4e14bffab9fb50d2fe8660f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:45.960871 10188 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.961767 9927 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.962962 13305 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.963250 13306 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.964715 10058 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.965314 13305 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.965289 9798 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.965991 12741 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.966934 10322 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.968120 9667 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:45.968652 13305 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.969048 13306 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:45.971738 13306 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.972306 9667 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.973165 10058 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.977380 10188 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.977749 10322 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.983115 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:45.983695 9389 catalog_manager.cc:5484] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.984186 9927 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:45.985898 9798 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:46.022626 13267 raft_consensus.cc:1050] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:46.023916 13267 consensus_queue.cc:237] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:46.027814 11958 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.028592 10322 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.029491 13266 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.031090 10188 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.031697 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.034965 13266 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.041376 9667 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.041931 10058 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.042217 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.043440 9535 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.043433 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.043987 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.045522 9798 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.049966 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.053474 12815 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.057276 9798 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.064175 9389 catalog_manager.cc:5484] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06f1b17731834350a289de93abab8542" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.068996 10188 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.075598 13349 ts_tablet_manager.cc:902] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:46.075956 9667 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.079670 11958 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.081728 9535 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.083693 10322 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.088730 10058 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.106621 9927 consensus_queue.cc:237] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:46.110064 11958 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:46.110729 9667 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:46.110977 10058 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:46.111421 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:46.111662 9535 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:46.112005 13267 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:46.113265 13267 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:46.113700 12815 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:46.117385 10322 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:46.117007 13267 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.119387 13331 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:46.119480 10058 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.120409 9535 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.127476 9392 catalog_manager.cc:5484] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "06f1b17731834350a289de93abab8542" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.129261 9381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd41cbc6c69441c1af5c36c382cfe366 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:46.129914 11958 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.133229 10169 tablet_service.cc:1514] Processing DeleteTablet for tablet dd41cbc6c69441c1af5c36c382cfe366 with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:46.136301 9798 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:46.137913 9667 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.138679 13267 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:46.144075 9798 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.148763 10322 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.161142 13355 tablet_replica.cc:330] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:46.161376 13355 raft_consensus.cc:2229] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:46.161876 13355 raft_consensus.cc:2258] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:46.171707 13355 ts_tablet_manager.cc:1875] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:46.175575 13349 tablet_copy_client.cc:308] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.7:42473
I20240503 08:14:46.179833 13355 ts_tablet_manager.cc:1888] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:46.180086 13355 log.cc:1196] T dd41cbc6c69441c1af5c36c382cfe366 P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/dd41cbc6c69441c1af5c36c382cfe366
I20240503 08:14:46.181149 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet dd41cbc6c69441c1af5c36c382cfe366 (table rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]) successfully deleted
I20240503 08:14:46.189246 10342 tablet_copy_service.cc:140] P 0cb18b8b10f944e39829faf16e061fd6: Received BeginTabletCopySession request for tablet fdb88bebfdab4e54abc5dba5834a3298 from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:39157)
I20240503 08:14:46.189435 10342 tablet_copy_service.cc:161] P 0cb18b8b10f944e39829faf16e061fd6: Beginning new tablet copy session on tablet fdb88bebfdab4e54abc5dba5834a3298 from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:39157: session id = ca5fa388d5b84a0abac1c05275155dcf-fdb88bebfdab4e54abc5dba5834a3298
I20240503 08:14:46.191844 10342 tablet_copy_source_session.cc:205] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:46.193447 13349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb88bebfdab4e54abc5dba5834a3298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:46.198069 13349 tablet_copy_client.cc:783] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:46.198315 13349 tablet_copy_client.cc:647] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:46.199959 13349 tablet_copy_client.cc:523] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:46.202206 13349 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:46.250180 13349 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:46.250631 13349 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:46.250963 13349 ts_tablet_manager.cc:1367] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.049s user 0.037s sys 0.010s
I20240503 08:14:46.251574 13349 raft_consensus.cc:348] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:46.251884 13349 raft_consensus.cc:729] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:46.252092 13349 consensus_queue.cc:260] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:46.252812 13349 ts_tablet_manager.cc:1398] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:46.253819 10342 tablet_copy_service.cc:342] P 0cb18b8b10f944e39829faf16e061fd6: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-fdb88bebfdab4e54abc5dba5834a3298 received from {username='slave'} at 127.24.123.8:39157
I20240503 08:14:46.253937 10342 tablet_copy_service.cc:433] P 0cb18b8b10f944e39829faf16e061fd6: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-fdb88bebfdab4e54abc5dba5834a3298 on tablet fdb88bebfdab4e54abc5dba5834a3298 with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:46.315371 11958 raft_consensus.cc:1203] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:46.365936 13349 ts_tablet_manager.cc:902] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:46.367169 13349 tablet_copy_client.cc:308] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:41805
I20240503 08:14:46.367799 9818 tablet_copy_service.cc:140] P 8de08e679a904ee48ba659861c730725: Received BeginTabletCopySession request for tablet 2a2867608f0a4d69a3e169a59e358d10 from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:35377)
I20240503 08:14:46.367942 9818 tablet_copy_service.cc:161] P 8de08e679a904ee48ba659861c730725: Beginning new tablet copy session on tablet 2a2867608f0a4d69a3e169a59e358d10 from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:35377: session id = ca5fa388d5b84a0abac1c05275155dcf-2a2867608f0a4d69a3e169a59e358d10
I20240503 08:14:46.370029 9818 tablet_copy_source_session.cc:205] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:46.371528 13349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2867608f0a4d69a3e169a59e358d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:46.376166 13349 tablet_copy_client.cc:783] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:46.376382 13349 tablet_copy_client.cc:647] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:46.378144 13349 tablet_copy_client.cc:523] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:46.380736 13349 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:46.418145 13349 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:46.418594 13349 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:46.418838 13349 ts_tablet_manager.cc:1367] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.038s user 0.033s sys 0.002s
I20240503 08:14:46.419526 13349 raft_consensus.cc:348] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:46.419782 13349 raft_consensus.cc:729] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:46.420035 13349 consensus_queue.cc:260] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:46.420598 13349 ts_tablet_manager.cc:1398] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:46.421391 9818 tablet_copy_service.cc:342] P 8de08e679a904ee48ba659861c730725: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-2a2867608f0a4d69a3e169a59e358d10 received from {username='slave'} at 127.24.123.8:35377
I20240503 08:14:46.421561 9818 tablet_copy_service.cc:433] P 8de08e679a904ee48ba659861c730725: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-2a2867608f0a4d69a3e169a59e358d10 on tablet 2a2867608f0a4d69a3e169a59e358d10 with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:46.439828 13349 ts_tablet_manager.cc:902] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:46.441433 13349 tablet_copy_client.cc:308] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:46.442070 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:50757)
I20240503 08:14:46.442237 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 3a9fcac8b4e14bffab9fb50d2fe8660f from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:50757: session id = ca5fa388d5b84a0abac1c05275155dcf-3a9fcac8b4e14bffab9fb50d2fe8660f
I20240503 08:14:46.444873 9555 tablet_copy_source_session.cc:205] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:46.446302 13349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:46.446465 10188 consensus_queue.cc:237] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } }
I20240503 08:14:46.451408 13349 tablet_copy_client.cc:783] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:46.451776 13349 tablet_copy_client.cc:647] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:46.453995 13349 tablet_copy_client.cc:523] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:46.456506 9798 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.456487 9667 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.457692 13295 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.458302 13295 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.459089 13349 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:46.466046 12224 raft_consensus.cc:1203] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:46.468717 9535 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.469424 13085 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.474181 10322 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.475333 12706 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.477891 10058 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.477923 9927 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.478544 13085 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.479180 13295 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.491412 13295 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:46.495883 9927 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:46.496403 9667 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:46.498152 9535 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:46.500321 9798 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:46.503705 9387 catalog_manager.cc:5484] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:46.506314 10058 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:46.506407 10322 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } }
I20240503 08:14:46.518460 10188 consensus_queue.cc:237] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:46.521744 10058 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:14:46.522053 10124 consensus_peers.cc:485] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9306edc6261425cb2c7bef9caf9135b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:46.522408 9535 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.522961 9665 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.523998 9926 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.524067 13085 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.524605 13085 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.525074 13085 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.525786 12706 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.525988 9798 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.527137 12706 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.529022 10322 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.529597 13295 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.534485 13085 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.539865 9535 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.539865 9798 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.545845 9389 catalog_manager.cc:5484] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.547677 13349 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:46.548105 13349 tablet_bootstrap.cc:492] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:46.548388 13349 ts_tablet_manager.cc:1367] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.089s user 0.043s sys 0.008s
I20240503 08:14:46.548621 9379 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c9306edc6261425cb2c7bef9caf9135b with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:46.549196 13349 raft_consensus.cc:348] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:46.551723 13349 raft_consensus.cc:729] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:46.551992 13349 consensus_queue.cc:260] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:46.553226 13349 ts_tablet_manager.cc:1398] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:14:46.553983 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-3a9fcac8b4e14bffab9fb50d2fe8660f received from {username='slave'} at 127.24.123.8:50757
I20240503 08:14:46.554121 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-3a9fcac8b4e14bffab9fb50d2fe8660f on tablet 3a9fcac8b4e14bffab9fb50d2fe8660f with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:46.564474 10322 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.565048 9667 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.570344 10058 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
W20240503 08:14:46.583005 10127 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:38565, user_credentials={real_user=slave}} blocked reactor thread for 52295us
I20240503 08:14:46.584479 9926 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.590039 10058 consensus_queue.cc:237] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:46.592962 10322 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.593071 10188 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.593166 9926 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.593557 13188 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.593858 13381 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.594661 13401 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.595435 9798 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.595832 9535 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.596489 9667 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:46.600425 13188 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.600953 13188 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.601611 13188 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.617445 13381 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:46.619233 10321 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:46.621910 9926 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:46.622251 9535 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:46.626138 10188 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:46.628232 9389 catalog_manager.cc:5484] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:46.634435 10058 consensus_queue.cc:237] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:46.635259 9667 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
W20240503 08:14:46.638036 9993 consensus_peers.cc:485] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7247b6f7e08946709b215d2f6a26a2a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:46.638645 9926 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.638963 9533 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.639230 13403 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.639521 10321 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.642517 10188 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.643775 13403 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.644349 13403 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.644680 13188 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.645750 9665 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.646437 13401 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:46.647498 9796 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:46.648418 13401 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:46.648903 13401 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.649479 9667 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.649812 9796 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:46.653079 9380 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7247b6f7e08946709b215d2f6a26a2a4 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:46.654515 9926 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.656185 9796 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.659916 9389 catalog_manager.cc:5484] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.665714 10321 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.666405 10189 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.674016 9535 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:46.825872 12224 raft_consensus.cc:1203] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:46.932718 13085 raft_consensus.cc:1050] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:46.933627 13085 consensus_queue.cc:237] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:46.936636 9926 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.937130 13279 raft_consensus.cc:1050] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:46.937291 9667 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.937381 10058 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.937865 13395 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.938452 10321 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.938287 13279 consensus_queue.cc:237] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:46.938570 9796 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.939908 11958 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.940429 13395 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.940932 13395 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.943476 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.943715 13295 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.944118 9535 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.946089 13393 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.947016 12224 raft_consensus.cc:1203] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:46.949860 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.976166 12224 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:46.977875 13059 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:46.981200 13458 ts_tablet_manager.cc:902] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:46.982616 13395 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.982702 9927 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.983122 10058 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.988894 10321 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.988627 11958 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.991536 9389 catalog_manager.cc:5484] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:46.995132 9667 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.001874 9665 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.004637 13366 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.008116 9533 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.008235 9798 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.010564 13366 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.011255 13366 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.025907 10189 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.026048 9927 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.027079 10058 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.027822 13366 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.028447 13366 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.034839 13366 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.045387 9533 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.048151 13279 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.048848 9798 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.052518 10060 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.053380 10188 consensus_queue.cc:237] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:47.054294 9796 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.054548 12224 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.055193 9926 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.059895 10321 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.060217 11958 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.061244 9385 catalog_manager.cc:5484] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf reported cstate change: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.062747 9797 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.065501 9665 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.065554 9535 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.067200 9533 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.066001 10060 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.068423 13085 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.069087 13085 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.069538 13085 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.069964 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.070969 13458 tablet_copy_client.cc:308] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.6:37175
I20240503 08:14:47.071605 10211 tablet_copy_service.cc:140] P fa60e870de844325bad4b9185eaee468: Received BeginTabletCopySession request for tablet c9306edc6261425cb2c7bef9caf9135b from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:48377)
I20240503 08:14:47.071772 10211 tablet_copy_service.cc:161] P fa60e870de844325bad4b9185eaee468: Beginning new tablet copy session on tablet c9306edc6261425cb2c7bef9caf9135b from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:48377: session id = 3a70e21b51a142bd8f7dc93fa58742fb-c9306edc6261425cb2c7bef9caf9135b
I20240503 08:14:47.073268 9665 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.074609 10211 tablet_copy_source_session.cc:205] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:47.082520 13295 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.083344 12706 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.086022 10188 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.090992 13395 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.091617 10321 consensus_queue.cc:237] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:47.095402 9379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 014f053652224d60ac8ef85064f0cabe with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:47.097833 12224 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.097565 10319 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.098063 9796 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.098737 13059 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.099613 13059 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.100766 9926 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.097214 11958 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.104977 13366 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.106937 9667 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.108728 10060 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.113725 10058 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.114289 13366 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.116482 9390 catalog_manager.cc:5484] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.118218 9665 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.118949 13458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9306edc6261425cb2c7bef9caf9135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:47.120277 13366 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.121212 9534 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.122650 9797 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:47.123432 13366 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.126269 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 014f053652224d60ac8ef85064f0cabe with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:47.126031 9665 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.131677 9385 catalog_manager.cc:5484] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.134124 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fdb88bebfdab4e54abc5dba5834a3298 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:47.134441 9926 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.139396 10169 tablet_service.cc:1514] Processing DeleteTablet for tablet fdb88bebfdab4e54abc5dba5834a3298 with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:47.145397 10060 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.150600 13476 tablet_replica.cc:330] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:47.150991 13476 raft_consensus.cc:2229] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:47.153273 13476 raft_consensus.cc:2258] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:47.154807 13476 ts_tablet_manager.cc:1875] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:47.160210 13478 tablet_replica.cc:330] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:47.160507 13478 raft_consensus.cc:2229] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:47.160852 13478 raft_consensus.cc:2258] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:47.161083 13458 tablet_copy_client.cc:783] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:47.161329 13458 tablet_copy_client.cc:647] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:47.162526 13478 ts_tablet_manager.cc:1875] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:47.163837 13458 tablet_copy_client.cc:523] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:47.164800 13476 ts_tablet_manager.cc:1888] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:47.164937 13476 log.cc:1196] T 014f053652224d60ac8ef85064f0cabe P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/014f053652224d60ac8ef85064f0cabe
I20240503 08:14:47.165671 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet 014f053652224d60ac8ef85064f0cabe (table rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]) successfully deleted
I20240503 08:14:47.167341 13479 ts_tablet_manager.cc:902] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:47.168864 13458 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:47.171368 13478 ts_tablet_manager.cc:1888] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:47.171568 13478 log.cc:1196] T fdb88bebfdab4e54abc5dba5834a3298 P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/fdb88bebfdab4e54abc5dba5834a3298
I20240503 08:14:47.172502 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet fdb88bebfdab4e54abc5dba5834a3298 (table rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]) successfully deleted
W20240503 08:14:47.173094 10255 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.3:41805, user_credentials={real_user=slave}} blocked reactor thread for 74472.5us
I20240503 08:14:47.177227 9534 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.177022 12224 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.178074 9797 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.183552 13477 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 08:14:47.190536 9535 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.223062 13479 tablet_copy_client.cc:308] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43643
I20240503 08:14:47.223994 10080 tablet_copy_service.cc:140] P a1402a75453d4ccf992606dcce77dc51: Received BeginTabletCopySession request for tablet 7247b6f7e08946709b215d2f6a26a2a4 from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:58265)
I20240503 08:14:47.224185 10080 tablet_copy_service.cc:161] P a1402a75453d4ccf992606dcce77dc51: Beginning new tablet copy session on tablet 7247b6f7e08946709b215d2f6a26a2a4 from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:58265: session id = 3a70e21b51a142bd8f7dc93fa58742fb-7247b6f7e08946709b215d2f6a26a2a4
I20240503 08:14:47.227948 10080 tablet_copy_source_session.cc:205] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:47.229609 13479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7247b6f7e08946709b215d2f6a26a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:47.238170 13479 tablet_copy_client.cc:783] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:47.238651 13479 tablet_copy_client.cc:647] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:47.240054 13458 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:47.240662 13458 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:47.240731 13479 tablet_copy_client.cc:523] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:47.241014 13458 ts_tablet_manager.cc:1367] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.073s user 0.037s sys 0.006s
I20240503 08:14:47.241803 13458 raft_consensus.cc:348] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:47.242496 13458 raft_consensus.cc:729] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:47.242710 13458 consensus_queue.cc:260] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:47.243360 13458 ts_tablet_manager.cc:1398] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:14:47.244150 13479 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:47.244218 10211 tablet_copy_service.cc:342] P fa60e870de844325bad4b9185eaee468: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-c9306edc6261425cb2c7bef9caf9135b received from {username='slave'} at 127.24.123.9:48377
I20240503 08:14:47.244341 10211 tablet_copy_service.cc:433] P fa60e870de844325bad4b9185eaee468: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-c9306edc6261425cb2c7bef9caf9135b on tablet c9306edc6261425cb2c7bef9caf9135b with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:47.288081 13479 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:47.288551 13479 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:47.290045 13479 ts_tablet_manager.cc:1367] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.045s user 0.034s sys 0.005s
I20240503 08:14:47.290990 13479 raft_consensus.cc:348] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:47.291330 13479 raft_consensus.cc:729] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:47.291550 13479 consensus_queue.cc:260] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:47.292300 13479 ts_tablet_manager.cc:1398] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:47.293062 10080 tablet_copy_service.cc:342] P a1402a75453d4ccf992606dcce77dc51: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-7247b6f7e08946709b215d2f6a26a2a4 received from {username='slave'} at 127.24.123.9:58265
I20240503 08:14:47.293198 10080 tablet_copy_service.cc:433] P a1402a75453d4ccf992606dcce77dc51: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-7247b6f7e08946709b215d2f6a26a2a4 on tablet 7247b6f7e08946709b215d2f6a26a2a4 with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:47.427337 13445 raft_consensus.cc:1050] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:47.428220 13445 consensus_queue.cc:237] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:47.431187 12224 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.431780 10188 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.431871 10321 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.432031 9926 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.432597 13396 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.432905 13106 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.433132 9797 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.433533 13106 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.434252 13396 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.436580 9667 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.436528 13396 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.437292 13396 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.437500 10060 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.438674 13396 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.456468 13396 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.460983 9797 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.463450 10060 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.463704 10188 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.464509 12224 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.469480 9667 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.475358 9389 catalog_manager.cc:5484] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.481884 10321 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.488896 9926 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.521095 9535 consensus_queue.cc:237] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:47.524158 10321 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:47.524152 10188 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:47.524780 10060 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:47.524890 13396 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:47.525131 9926 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:47.525354 9797 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:47.525408 13445 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:47.525911 13515 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:47.526332 13396 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:47.526597 13523 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:47.531870 9667 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:47.532577 13396 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:47.532441 13523 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:47.533830 9926 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:47.538107 10321 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:47.538378 10188 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:47.543339 13296 raft_consensus.cc:1050] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:47.543787 10060 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:47.544515 13296 consensus_queue.cc:237] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:47.545593 9391 catalog_manager.cc:5484] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:47.552386 9535 consensus_queue.cc:237] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
W20240503 08:14:47.556622 9470 consensus_peers.cc:485] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cb26d36a69d4f4ab43401822096756f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:47.557978 11958 raft_consensus.cc:1203] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:47.560922 10060 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:47.561354 10188 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:47.561698 13396 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:47.562322 13445 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:47.565371 10322 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:47.566205 13523 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:47.569386 13445 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:47.571352 12224 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.571974 12949 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.573246 10188 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.574201 12949 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.576014 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8cb26d36a69d4f4ab43401822096756f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:47.578102 9929 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.577744 10322 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:47.578493 10321 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.578925 12949 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.579695 13537 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.582000 10058 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.583842 9385 catalog_manager.cc:5484] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:47.588243 10189 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:47.591203 13296 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.592362 9535 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.593242 9796 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:47.593832 13539 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.596284 10060 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:47.599416 9667 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:47.604333 9927 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:47.605090 13445 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:47.606937 9665 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:47.607551 13523 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:47.609028 9926 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:47.609151 9665 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:47.615841 9665 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:47.616869 13523 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:47.619117 13543 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:47.619323 9796 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:47.623891 13539 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.624737 9535 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.629447 9796 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:47.630537 9391 catalog_manager.cc:5484] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.635504 10058 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.638583 9929 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.640753 10188 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.640888 12224 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.641319 10321 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.648873 9665 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.651885 11958 raft_consensus.cc:1203] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:47.664418 9796 consensus_queue.cc:237] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:47.668021 12224 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.668141 10321 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.668715 13554 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.668987 13553 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.670296 9535 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.670830 13554 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.671023 9665 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.671653 13542 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.673655 10188 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.674832 13539 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.674497 13537 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.676481 10188 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.678408 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2a2867608f0a4d69a3e169a59e358d10 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:47.678962 12224 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.679579 9665 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.679939 9385 catalog_manager.cc:5484] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:47.683382 10321 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.687446 9535 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.692390 10060 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.693249 13537 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.696375 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2867608f0a4d69a3e169a59e358d10 with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:47.695039 10060 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.707363 13556 tablet_replica.cc:330] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:47.707602 13556 raft_consensus.cc:2229] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:47.708007 13556 raft_consensus.cc:2258] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:47.709041 13556 ts_tablet_manager.cc:1875] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:47.714228 13556 ts_tablet_manager.cc:1888] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:47.714344 13556 log.cc:1196] T 2a2867608f0a4d69a3e169a59e358d10 P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/2a2867608f0a4d69a3e169a59e358d10
I20240503 08:14:47.715015 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet 2a2867608f0a4d69a3e169a59e358d10 (table rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]) successfully deleted
I20240503 08:14:47.967154 9535 consensus_queue.cc:237] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:47.970475 12224 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.970907 9926 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.970896 10321 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.971611 13445 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.971906 13523 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.972597 13396 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.974885 9796 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.975554 13106 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.976336 13514 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.979552 9665 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.979805 10060 raft_consensus.cc:1261] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:47.979964 9796 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } }
I20240503 08:14:47.980396 13523 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.981756 9926 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.982920 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:47.980665 9797 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.984341 13523 consensus_queue.cc:1035] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:47.986070 9665 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.986529 9392 catalog_manager.cc:5484] T 3a9fcac8b4e14bffab9fb50d2fe8660f P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:47.987677 10321 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.990554 12224 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:47.994730 10321 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:47.994768 10060 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:47.995525 13539 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:47.995677 10169 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a9fcac8b4e14bffab9fb50d2fe8660f with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:47.996685 13543 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:47.996939 10188 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:47.998116 13543 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:48.001077 9667 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:48.001721 13543 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:48.005527 9535 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:48.006045 13584 tablet_replica.cc:330] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:48.006121 13543 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:48.006328 13584 raft_consensus.cc:2229] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:48.006412 9926 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:14:48.006680 13584 raft_consensus.cc:2258] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:48.007028 13543 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:48.019188 10058 raft_consensus.cc:2932] T 3a9fcac8b4e14bffab9fb50d2fe8660f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:48.020473 13584 ts_tablet_manager.cc:1875] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:48.028395 13543 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:48.033113 10059 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:48.035130 9533 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:48.035243 9667 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:48.042433 13584 ts_tablet_manager.cc:1888] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:48.042433 9389 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:48.043494 13584 log.cc:1196] T 3a9fcac8b4e14bffab9fb50d2fe8660f P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/3a9fcac8b4e14bffab9fb50d2fe8660f
I20240503 08:14:48.044493 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet 3a9fcac8b4e14bffab9fb50d2fe8660f (table rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]) successfully deleted
I20240503 08:14:48.048314 10189 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:48.050794 9796 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:48.052191 10321 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:48.054692 12706 raft_consensus.cc:1050] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:48.056411 12706 consensus_queue.cc:237] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:48.059319 9533 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:48.060397 13542 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:48.062935 10188 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:48.063119 10321 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:48.063822 13296 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:48.064651 13539 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:48.078500 9926 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:48.079274 13543 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:14:48.080734 9927 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } }
I20240503 08:14:48.087575 13296 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:48.090288 10188 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:48.090564 9926 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:48.096163 13597 ts_tablet_manager.cc:902] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:48.097110 9385 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:48.101195 9378 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 49a60d2ed0744f35be8054955dcf8aa0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:14:48.102303 9533 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:48.103235 11958 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.104148 13295 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.106012 9667 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.106637 13295 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.109076 10321 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:48.114308 10322 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.114809 10060 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.115483 13295 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.115875 9926 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.116053 13295 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.116384 13395 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.120729 9533 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.123634 9665 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:48.123910 10059 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:48.124095 9796 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.125100 13395 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.125195 13539 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:48.125633 13395 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.125728 13539 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:14:48.140420 10059 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:48.150457 9667 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
W20240503 08:14:48.159693 9733 consensus_peers.cc:485] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49a60d2ed0744f35be8054955dcf8aa0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:48.167335 13395 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.167922 10322 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.171452 11958 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.172073 9796 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.174129 9392 catalog_manager.cc:5484] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.177196 9533 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.179350 10060 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.183388 10189 consensus_queue.cc:237] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:48.188231 13401 raft_consensus.cc:1050] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:48.189491 13401 consensus_queue.cc:237] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:48.209124 11957 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.210922 13555 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.210846 9667 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.223331 11957 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.224025 13395 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.223783 9927 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.225869 10060 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.227247 13395 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.227875 10189 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.229722 9534 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.229776 9796 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.230469 13615 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.233533 9535 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.234321 13619 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.235004 13619 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.235033 13395 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.237334 12706 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.237849 9535 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.240583 9664 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.241593 13617 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.241919 9379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9306edc6261425cb2c7bef9caf9135b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:48.243520 10060 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.243733 11957 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.249653 9798 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.249926 9385 catalog_manager.cc:5484] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.251145 13395 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.253731 9797 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.255337 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet c9306edc6261425cb2c7bef9caf9135b with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:48.262892 10321 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.265239 9665 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.266448 13395 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.269891 13619 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.269778 9667 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.272423 13626 tablet_replica.cc:330] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:48.272715 13626 raft_consensus.cc:2229] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:48.273075 13626 raft_consensus.cc:2258] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:48.274097 13626 ts_tablet_manager.cc:1875] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:48.275920 11957 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
W20240503 08:14:48.279983 9996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:38565, user_credentials={real_user=slave}} blocked reactor thread for 67959.5us
I20240503 08:14:48.280858 9927 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:48.275182 13617 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.282094 13555 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:48.282897 9385 catalog_manager.cc:5484] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.289752 10322 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.290431 13624 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.292232 10319 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.297374 13626 ts_tablet_manager.cc:1888] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:48.299168 10189 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.302742 9796 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.304594 9665 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.308162 9534 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.310072 13626 log.cc:1196] T c9306edc6261425cb2c7bef9caf9135b P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/c9306edc6261425cb2c7bef9caf9135b
I20240503 08:14:48.316198 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet c9306edc6261425cb2c7bef9caf9135b (table rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]) successfully deleted
I20240503 08:14:48.316183 9927 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.316329 10321 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
W20240503 08:14:48.329530 10255 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.1:39353, user_credentials={real_user=slave}} blocked reactor thread for 131168us
I20240503 08:14:48.338248 13597 tablet_copy_client.cc:308] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:48.339032 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 8cb26d36a69d4f4ab43401822096756f from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:59753)
I20240503 08:14:48.339228 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 8cb26d36a69d4f4ab43401822096756f from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:59753: session id = 925c339b047b4cf5bae840c5a357d3de-8cb26d36a69d4f4ab43401822096756f
I20240503 08:14:48.342344 9555 tablet_copy_source_session.cc:205] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:48.344069 13597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb26d36a69d4f4ab43401822096756f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.351722 13597 tablet_copy_client.cc:783] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:48.351999 13597 tablet_copy_client.cc:647] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:48.354209 13597 tablet_copy_client.cc:523] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:48.357152 13597 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:48.395056 13597 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:48.395506 13597 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:48.395769 13597 ts_tablet_manager.cc:1367] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.009s
I20240503 08:14:48.396472 13597 raft_consensus.cc:348] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:48.396764 13597 raft_consensus.cc:729] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:48.396972 13597 consensus_queue.cc:260] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:48.402133 13597 ts_tablet_manager.cc:1398] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240503 08:14:48.403720 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-8cb26d36a69d4f4ab43401822096756f received from {username='slave'} at 127.24.123.10:59753
I20240503 08:14:48.403860 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-8cb26d36a69d4f4ab43401822096756f on tablet 8cb26d36a69d4f4ab43401822096756f with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:48.499943 13597 ts_tablet_manager.cc:902] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:48.501081 13597 tablet_copy_client.cc:308] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:41805
I20240503 08:14:48.506600 9818 tablet_copy_service.cc:140] P 8de08e679a904ee48ba659861c730725: Received BeginTabletCopySession request for tablet 49a60d2ed0744f35be8054955dcf8aa0 from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:51097)
I20240503 08:14:48.506778 9818 tablet_copy_service.cc:161] P 8de08e679a904ee48ba659861c730725: Beginning new tablet copy session on tablet 49a60d2ed0744f35be8054955dcf8aa0 from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:51097: session id = 925c339b047b4cf5bae840c5a357d3de-49a60d2ed0744f35be8054955dcf8aa0
I20240503 08:14:48.508816 9818 tablet_copy_source_session.cc:205] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:48.509999 13597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.514540 13597 tablet_copy_client.cc:783] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:48.514735 13597 tablet_copy_client.cc:647] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:48.516259 13597 tablet_copy_client.cc:523] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:48.518471 13597 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:48.551065 13597 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:48.551465 13597 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:48.551703 13597 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.033s user 0.034s sys 0.000s
I20240503 08:14:48.552403 13597 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:48.552637 13597 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:48.552842 13597 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:48.553359 13597 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:48.554078 9818 tablet_copy_service.cc:342] P 8de08e679a904ee48ba659861c730725: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-49a60d2ed0744f35be8054955dcf8aa0 received from {username='slave'} at 127.24.123.10:51097
I20240503 08:14:48.554214 9818 tablet_copy_service.cc:433] P 8de08e679a904ee48ba659861c730725: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-49a60d2ed0744f35be8054955dcf8aa0 on tablet 49a60d2ed0744f35be8054955dcf8aa0 with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:48.555953 12090 raft_consensus.cc:1203] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:48.796125 10060 consensus_queue.cc:237] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:48.798823 10322 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.798897 9534 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.798897 9926 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.799552 11958 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.799510 13616 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.799630 9667 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.800019 13616 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.800261 9796 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:48.800529 13616 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.800786 13555 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.801268 13617 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.801729 13615 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:48.803491 13615 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.804491 10322 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.804765 9667 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.806535 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7247b6f7e08946709b215d2f6a26a2a4 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:48.807650 9926 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.808646 9534 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.811123 11958 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.814100 9392 catalog_manager.cc:5484] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:48.818461 9796 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:48.833190 10169 tablet_service.cc:1514] Processing DeleteTablet for tablet 7247b6f7e08946709b215d2f6a26a2a4 with delete_type TABLET_DATA_TOMBSTONED (TS fa60e870de844325bad4b9185eaee468 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53340
I20240503 08:14:48.833827 13679 tablet_replica.cc:330] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468: stopping tablet replica
I20240503 08:14:48.834110 13679 raft_consensus.cc:2229] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:48.834424 13679 raft_consensus.cc:2258] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:48.835644 13679 ts_tablet_manager.cc:1875] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:48.842183 13679 ts_tablet_manager.cc:1888] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:48.842370 13679 log.cc:1196] T 7247b6f7e08946709b215d2f6a26a2a4 P fa60e870de844325bad4b9185eaee468: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-5/wal/wals/7247b6f7e08946709b215d2f6a26a2a4
I20240503 08:14:48.843170 9379 catalog_manager.cc:4830] TS fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): tablet 7247b6f7e08946709b215d2f6a26a2a4 (table rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]) successfully deleted
I20240503 08:14:48.895514 12090 raft_consensus.cc:1203] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:14:49.117759 13523 raft_consensus.cc:1050] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:49.118862 13523 consensus_queue.cc:237] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:49.122424 12090 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.122442 10189 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.123056 13514 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.123567 10060 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.123890 9926 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.124718 10322 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.124826 13514 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.125362 13514 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.125675 13708 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.128602 13708 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.134719 9667 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.134742 9796 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.135458 13514 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.135416 13708 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.136014 13523 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.140059 12090 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.143159 10189 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.145285 9667 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.145413 10060 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.149039 9387 catalog_manager.cc:5484] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de reported cstate change: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.150499 9926 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.154805 10322 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.167239 9796 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.353014 13542 raft_consensus.cc:1050] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:49.354290 13542 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:49.357177 10322 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.357156 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.357985 13537 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.358141 12090 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.358523 13537 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.359007 13543 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.361304 10060 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.361889 13695 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.365479 9926 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.365643 9534 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.366055 9667 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:49.366130 13695 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.366683 13695 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.366932 13542 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:49.372501 13543 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.378432 10060 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.379674 12090 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.381821 9667 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.383072 9534 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.383960 10189 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.385475 9389 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de reported cstate change: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.381143 9926 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.388037 10322 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.398197 9796 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:49.402159 10060 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.402204 12090 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.402822 13729 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.403231 13537 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.403602 10322 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.404125 13537 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.407058 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.407140 9534 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.407361 9667 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.407577 13695 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.407912 13542 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.408340 13537 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.411187 13537 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.412408 10188 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.414211 9534 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.416095 9667 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.417236 9391 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.421319 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:49.422120 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49a60d2ed0744f35be8054955dcf8aa0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:49.422224 10322 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.422230 12090 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.427781 10060 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.437705 13737 tablet_replica.cc:330] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:49.437894 13737 raft_consensus.cc:2229] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:49.438212 13737 raft_consensus.cc:2258] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.438980 13737 ts_tablet_manager.cc:1875] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:49.443135 13737 ts_tablet_manager.cc:1888] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:49.443260 13737 log.cc:1196] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/49a60d2ed0744f35be8054955dcf8aa0
I20240503 08:14:49.443902 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet 49a60d2ed0744f35be8054955dcf8aa0 (table rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]) successfully deleted
I20240503 08:14:49.617978 9534 consensus_queue.cc:237] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:49.620687 10322 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.620695 10060 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.620760 12090 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.621245 13523 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.621374 9667 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.621544 13717 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.621731 10189 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.621886 13708 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.622390 13708 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.622798 9796 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:49.622763 13514 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.624317 13514 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:49.627091 13716 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.628731 9796 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.628849 12090 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.629740 10060 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.631210 9667 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.634598 9388 catalog_manager.cc:5484] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.635744 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cb26d36a69d4f4ab43401822096756f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:49.636256 10189 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.636536 10322 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:49.645630 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb26d36a69d4f4ab43401822096756f with delete_type TABLET_DATA_TOMBSTONED (TS 06f1b17731834350a289de93abab8542 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53556
I20240503 08:14:49.646080 13737 tablet_replica.cc:330] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542: stopping tablet replica
I20240503 08:14:49.646281 13737 raft_consensus.cc:2229] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:49.646510 13737 raft_consensus.cc:2258] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.648453 13737 ts_tablet_manager.cc:1875] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:49.653420 13737 ts_tablet_manager.cc:1888] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:14:49.653540 13737 log.cc:1196] T 8cb26d36a69d4f4ab43401822096756f P 06f1b17731834350a289de93abab8542: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-3/wal/wals/8cb26d36a69d4f4ab43401822096756f
I20240503 08:14:49.654225 9381 catalog_manager.cc:4830] TS 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): tablet 8cb26d36a69d4f4ab43401822096756f (table rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]) successfully deleted
I20240503 08:14:50.078428 9909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:50.107379 11938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.108525 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:14:50.179116 9513 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:50.188263 12070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.199329 9777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:50.220898 10038 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:50.222726 12204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.227360 10169 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:50.250250 9646 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:50.500635 9909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:50.516928 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:14:50.544064 12070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.555208 9777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:50.558841 9513 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:50.561012 10038 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:50.598775 9646 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:50.607513 11938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.607549 12204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:50.679728 10169 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:50.772702 9535 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:50.776046 12223 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.777194 13716 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.777529 9797 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.779736 13716 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:50.787320 9468 consensus_peers.cc:485] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fe97b80e1294c4791560daf991bfcdb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:50.788988 9665 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.788987 10189 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.789570 10321 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.789570 10060 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.790231 13849 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.790604 13523 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.791136 13523 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.791640 13716 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.805320 13523 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:50.808220 10189 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:50.813649 10060 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:50.813637 9796 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:50.818632 9389 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:50.826696 10321 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:50.826923 12223 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:50.833122 9665 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:50.895465 10060 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:50.898785 12223 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.899986 13858 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.901146 9927 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.901145 10321 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.901734 9665 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.902287 9535 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.902278 13871 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.902501 13616 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.903192 9796 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.904026 13871 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.904670 13871 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.905208 13871 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:50.910248 9995 consensus_peers.cc:485] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f6758eb87b047d29a060fcbfb775903. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:50.910384 13871 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:50.912802 9535 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:50.915907 12223 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:50.916044 9926 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:50.922411 9391 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf reported cstate change: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:50.929286 9796 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:50.936300 10321 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:50.936295 9665 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:50.967869 10189 consensus_queue.cc:237] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:50.971422 10321 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.971449 11958 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.972105 13395 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.972491 13801 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:50.979396 10124 consensus_peers.cc:485] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 014f053652224d60ac8ef85064f0cabe. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:50.984596 10060 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.984797 9796 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.984918 9535 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.985373 13395 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.985657 13801 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.985942 13886 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.987311 9665 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:50.987977 13801 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:50.991721 13395 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:50.992276 9665 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.000716 9388 catalog_manager.cc:5484] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.001277 10321 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:50.998490 11958 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.016044 9796 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.016922 10060 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.017259 9534 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.033552 9534 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:51.036195 12090 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.036237 9796 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.036442 10321 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.036929 13863 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.037454 9926 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.037442 13863 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.037954 13523 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:51.038192 9468 consensus_peers.cc:485] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1be499fcc2bf4716b16a5983950d3ed8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:51.039160 13523 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.039812 10060 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.043017 13523 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.048498 9665 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.049064 13523 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.056025 13523 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.056896 9796 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.063452 9391 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.067902 9926 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.067902 12090 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.080998 10059 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.086313 10321 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.089216 9665 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.112048 10321 consensus_queue.cc:237] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:51.115389 12223 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:14:51.115954 10257 consensus_peers.cc:485] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdb88bebfdab4e54abc5dba5834a3298. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:51.116005 13917 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.116652 9796 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.117203 13917 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.118757 9665 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.119513 13917 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.119874 9926 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.121248 13474 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.133229 9535 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.134766 13918 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.137610 13474 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.143321 9796 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.150354 9387 catalog_manager.cc:5484] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.151477 9535 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.151531 12223 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.154850 10059 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.155499 13917 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.155596 9665 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.155615 9927 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.158396 10059 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.179932 9535 consensus_queue.cc:237] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:51.183032 12090 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.183516 9927 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.183617 13863 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.183905 10059 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.184064 9665 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.184172 13863 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:51.184938 9468 consensus_peers.cc:485] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9428100e4a364805863419b22ca3c682. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:51.185043 13863 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.185410 13523 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.185587 9796 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.186228 10321 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.188012 13902 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.188491 13523 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.190485 13937 ts_tablet_manager.cc:902] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:51.193027 13523 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.194502 10060 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.199379 9927 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.200062 12090 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.200067 9796 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.204519 9665 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.204954 9391 catalog_manager.cc:5484] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.207262 10321 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.223205 13937 tablet_copy_client.cc:308] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:51.250628 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 3fe97b80e1294c4791560daf991bfcdb from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:55179)
I20240503 08:14:51.250838 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 3fe97b80e1294c4791560daf991bfcdb from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:55179: session id = 3a70e21b51a142bd8f7dc93fa58742fb-3fe97b80e1294c4791560daf991bfcdb
I20240503 08:14:51.253388 9555 tablet_copy_source_session.cc:205] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:51.254765 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:51.260381 9665 consensus_queue.cc:237] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:51.260635 13937 tablet_copy_client.cc:783] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:51.260916 13937 tablet_copy_client.cc:647] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:51.263080 13937 tablet_copy_client.cc:523] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:51.264250 12090 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.266675 13753 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.267710 13937 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:51.275933 9796 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.276475 10189 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.276937 10060 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.277232 13753 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.277547 13952 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.284929 10321 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.285106 13753 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.285542 9535 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.295199 13753 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.303467 13753 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:51.304769 9600 consensus_peers.cc:485] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8506211351284b7a8e2a166b8083977a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:51.305225 13952 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.312258 10322 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.318562 10189 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.319824 9387 catalog_manager.cc:5484] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.327384 13937 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:51.327780 13937 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:51.328058 13937 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.060s user 0.042s sys 0.009s
I20240503 08:14:51.326463 12090 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.328847 13937 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:51.329357 13937 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:51.329577 13937 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:51.330161 13937 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:51.334854 9797 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.336879 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-3fe97b80e1294c4791560daf991bfcdb received from {username='slave'} at 127.24.123.9:55179
I20240503 08:14:51.336989 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-3fe97b80e1294c4791560daf991bfcdb on tablet 3fe97b80e1294c4791560daf991bfcdb with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:51.343058 10059 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.345371 9534 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.359546 10189 consensus_queue.cc:237] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:51.362725 11958 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.362784 10059 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:14:51.363147 10126 consensus_peers.cc:485] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc9f0afdc01d4d03917b9040c2053426. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:51.363494 13395 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.363662 9796 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.364075 13886 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.364392 13395 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.367516 10321 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.368211 13886 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.369884 9665 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.370079 9534 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.370671 13969 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.371076 13395 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.373330 13395 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.377418 10321 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.378918 11958 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.379526 10059 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.380759 9796 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.384025 9387 catalog_manager.cc:5484] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:51.388948 9665 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.407157 9534 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.417951 10189 consensus_queue.cc:237] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:51.421011 12090 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.421478 9665 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.421530 10059 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.422027 13886 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.422637 9534 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.422771 9796 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.423424 10321 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.423339 13969 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:51.424233 10124 consensus_peers.cc:485] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a61f59d188824bf3b610771b2fc05f06. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:51.424635 13801 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.425099 13801 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.425561 13801 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.425949 13890 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.444139 13801 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.444792 9665 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.449290 10322 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.449826 9385 catalog_manager.cc:5484] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:51.450012 12090 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.478377 9796 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.473889 9534 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.478379 10059 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:51.494014 13993 ts_tablet_manager.cc:902] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:51.518728 13998 ts_tablet_manager.cc:902] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:51.524668 9534 consensus_queue.cc:237] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:51.526121 13995 ts_tablet_manager.cc:902] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:51.532043 13993 tablet_copy_client.cc:308] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.6:37175
W20240503 08:14:51.536339 9468 consensus_peers.cc:485] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cb26d36a69d4f4ab43401822096756f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:51.536413 12090 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.536687 10189 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.537144 13716 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.537545 13523 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.537721 10322 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.538393 13716 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.538530 10059 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.538758 9796 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.539896 13853 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.540493 13853 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.541357 9665 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.547803 13716 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.551144 10211 tablet_copy_service.cc:140] P fa60e870de844325bad4b9185eaee468: Received BeginTabletCopySession request for tablet 014f053652224d60ac8ef85064f0cabe from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:40727)
I20240503 08:14:51.551323 10211 tablet_copy_service.cc:161] P fa60e870de844325bad4b9185eaee468: Beginning new tablet copy session on tablet 014f053652224d60ac8ef85064f0cabe from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:40727: session id = ca5fa388d5b84a0abac1c05275155dcf-014f053652224d60ac8ef85064f0cabe
I20240503 08:14:51.551995 13937 ts_tablet_manager.cc:902] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:51.554440 10211 tablet_copy_source_session.cc:205] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:51.560276 13993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014f053652224d60ac8ef85064f0cabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:51.567867 13863 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.569046 9665 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.569020 10189 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.574180 10059 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.577574 9385 catalog_manager.cc:5484] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:51.579531 9796 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.591490 12089 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.593879 13998 tablet_copy_client.cc:308] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.7:42473
I20240503 08:14:51.594779 10342 tablet_copy_service.cc:140] P 0cb18b8b10f944e39829faf16e061fd6: Received BeginTabletCopySession request for tablet fdb88bebfdab4e54abc5dba5834a3298 from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:38815)
I20240503 08:14:51.594966 10342 tablet_copy_service.cc:161] P 0cb18b8b10f944e39829faf16e061fd6: Beginning new tablet copy session on tablet fdb88bebfdab4e54abc5dba5834a3298 from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:38815: session id = 925c339b047b4cf5bae840c5a357d3de-fdb88bebfdab4e54abc5dba5834a3298
I20240503 08:14:51.597877 10342 tablet_copy_source_session.cc:205] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:51.573295 13993 tablet_copy_client.cc:783] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:51.600654 13993 tablet_copy_client.cc:647] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:51.600767 13998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb88bebfdab4e54abc5dba5834a3298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:51.599359 10322 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:51.609050 13995 tablet_copy_client.cc:308] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43643
I20240503 08:14:51.616168 13993 tablet_copy_client.cc:523] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:51.624025 13993 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:51.628175 10080 tablet_copy_service.cc:140] P a1402a75453d4ccf992606dcce77dc51: Received BeginTabletCopySession request for tablet 0f6758eb87b047d29a060fcbfb775903 from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:44419)
I20240503 08:14:51.628346 10080 tablet_copy_service.cc:161] P a1402a75453d4ccf992606dcce77dc51: Beginning new tablet copy session on tablet 0f6758eb87b047d29a060fcbfb775903 from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:44419: session id = 925c339b047b4cf5bae840c5a357d3de-0f6758eb87b047d29a060fcbfb775903
I20240503 08:14:51.631163 10080 tablet_copy_source_session.cc:205] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:51.632099 13998 tablet_copy_client.cc:783] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:51.632516 13998 tablet_copy_client.cc:647] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:51.632758 13995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:51.634982 13998 tablet_copy_client.cc:523] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:51.639854 13998 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:51.643258 13995 tablet_copy_client.cc:783] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:51.644243 13995 tablet_copy_client.cc:647] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:51.646715 13995 tablet_copy_client.cc:523] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:51.650139 13995 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:51.680081 13937 tablet_copy_client.cc:308] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:51.680824 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 1be499fcc2bf4716b16a5983950d3ed8 from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:55179)
I20240503 08:14:51.680997 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 1be499fcc2bf4716b16a5983950d3ed8 from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:55179: session id = 3a70e21b51a142bd8f7dc93fa58742fb-1be499fcc2bf4716b16a5983950d3ed8
I20240503 08:14:51.683202 9555 tablet_copy_source_session.cc:205] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:51.690646 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:51.702771 13937 tablet_copy_client.cc:783] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:51.704910 13937 tablet_copy_client.cc:647] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:51.704653 14032 ts_tablet_manager.cc:902] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123)
I20240503 08:14:51.706857 13937 tablet_copy_client.cc:523] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:51.709719 13937 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:51.719151 9796 consensus_queue.cc:237] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:51.746981 13993 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:51.747440 13993 tablet_bootstrap.cc:492] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:51.747725 13993 ts_tablet_manager.cc:1367] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.124s user 0.048s sys 0.006s
I20240503 08:14:51.748521 13993 raft_consensus.cc:348] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:51.748874 13993 raft_consensus.cc:729] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:51.749082 13993 consensus_queue.cc:260] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
W20240503 08:14:51.753162 9733 consensus_peers.cc:485] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a2867608f0a4d69a3e169a59e358d10. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:51.754354 14032 tablet_copy_client.cc:308] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.2:36123
I20240503 08:14:51.755030 12223 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.755645 13800 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.767474 10189 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.768841 13800 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.776980 13995 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:51.783576 11958 raft_consensus.cc:1203] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:51.788372 13998 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:51.791203 13995 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:51.791545 13995 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.142s user 0.049s sys 0.003s
I20240503 08:14:51.792407 13995 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:51.792805 13995 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:51.793035 13995 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:51.793363 10322 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.793671 13995 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:14:51.794314 10080 tablet_copy_service.cc:342] P a1402a75453d4ccf992606dcce77dc51: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-0f6758eb87b047d29a060fcbfb775903 received from {username='slave'} at 127.24.123.10:44419
I20240503 08:14:51.794446 10080 tablet_copy_service.cc:433] P a1402a75453d4ccf992606dcce77dc51: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-0f6758eb87b047d29a060fcbfb775903 on tablet 0f6758eb87b047d29a060fcbfb775903 with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:51.795599 9687 tablet_copy_service.cc:140] P d1eb5e69c6264a20a6e8297cc5deb8d1: Received BeginTabletCopySession request for tablet 8506211351284b7a8e2a166b8083977a from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:51411)
I20240503 08:14:51.795759 9687 tablet_copy_service.cc:161] P d1eb5e69c6264a20a6e8297cc5deb8d1: Beginning new tablet copy session on tablet 8506211351284b7a8e2a166b8083977a from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:51411: session id = 3a70e21b51a142bd8f7dc93fa58742fb-8506211351284b7a8e2a166b8083977a
I20240503 08:14:51.797758 13800 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.799717 9687 tablet_copy_source_session.cc:205] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:51.800529 13998 tablet_bootstrap.cc:492] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:51.800846 13998 ts_tablet_manager.cc:1367] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.161s user 0.073s sys 0.023s
I20240503 08:14:51.801283 14032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8506211351284b7a8e2a166b8083977a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:51.801596 13998 raft_consensus.cc:348] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:51.801939 13998 raft_consensus.cc:729] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:51.802165 13998 consensus_queue.cc:260] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:51.802799 13998 ts_tablet_manager.cc:1398] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:14:51.803388 10342 tablet_copy_service.cc:342] P 0cb18b8b10f944e39829faf16e061fd6: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-fdb88bebfdab4e54abc5dba5834a3298 received from {username='slave'} at 127.24.123.10:38815
I20240503 08:14:51.803529 10342 tablet_copy_service.cc:433] P 0cb18b8b10f944e39829faf16e061fd6: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-fdb88bebfdab4e54abc5dba5834a3298 on tablet fdb88bebfdab4e54abc5dba5834a3298 with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:51.811808 14032 tablet_copy_client.cc:783] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:51.812243 14032 tablet_copy_client.cc:647] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:51.812789 13993 ts_tablet_manager.cc:1398] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.065s user 0.032s sys 0.028s
I20240503 08:14:51.814090 10059 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.814570 9667 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.814683 10211 tablet_copy_service.cc:342] P fa60e870de844325bad4b9185eaee468: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-014f053652224d60ac8ef85064f0cabe received from {username='slave'} at 127.24.123.8:40727
I20240503 08:14:51.814811 10211 tablet_copy_service.cc:433] P fa60e870de844325bad4b9185eaee468: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-014f053652224d60ac8ef85064f0cabe on tablet 014f053652224d60ac8ef85064f0cabe with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:51.815670 13800 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.816740 14032 tablet_copy_client.cc:523] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:51.819046 14017 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.820127 14032 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:51.836230 9534 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.836853 14017 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.861358 14035 ts_tablet_manager.cc:902] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:51.882905 14032 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:51.883524 14032 tablet_bootstrap.cc:492] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:51.883826 14032 ts_tablet_manager.cc:1367] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.064s user 0.051s sys 0.006s
I20240503 08:14:51.884635 14032 raft_consensus.cc:348] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:51.884981 14032 raft_consensus.cc:729] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:51.885234 14032 consensus_queue.cc:260] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:51.888406 14017 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.890682 14032 ts_tablet_manager.cc:1398] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:14:51.892314 10321 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.894521 9687 tablet_copy_service.cc:342] P d1eb5e69c6264a20a6e8297cc5deb8d1: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-8506211351284b7a8e2a166b8083977a received from {username='slave'} at 127.24.123.9:51411
I20240503 08:14:51.894645 9687 tablet_copy_service.cc:433] P d1eb5e69c6264a20a6e8297cc5deb8d1: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-8506211351284b7a8e2a166b8083977a on tablet 8506211351284b7a8e2a166b8083977a with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:51.898401 9387 catalog_manager.cc:5484] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.906701 10060 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.915620 9535 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.929193 13998 ts_tablet_manager.cc:902] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:51.939390 10319 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:51.944730 10188 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.967361 12223 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:51.973488 13998 tablet_copy_client.cc:308] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.6:37175
I20240503 08:14:51.974257 10211 tablet_copy_service.cc:140] P fa60e870de844325bad4b9185eaee468: Received BeginTabletCopySession request for tablet cc9f0afdc01d4d03917b9040c2053426 from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:34373)
I20240503 08:14:51.974486 10211 tablet_copy_service.cc:161] P fa60e870de844325bad4b9185eaee468: Beginning new tablet copy session on tablet cc9f0afdc01d4d03917b9040c2053426 from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:34373: session id = 925c339b047b4cf5bae840c5a357d3de-cc9f0afdc01d4d03917b9040c2053426
I20240503 08:14:51.977604 10211 tablet_copy_source_session.cc:205] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:51.978909 13998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f0afdc01d4d03917b9040c2053426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:51.985924 13998 tablet_copy_client.cc:783] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:51.986160 13998 tablet_copy_client.cc:647] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:51.989411 12089 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.990002 13917 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:51.997514 10059 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:51.998167 13917 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.000404 13998 tablet_copy_client.cc:523] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:52.004567 13998 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:52.004338 9667 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:52.011428 13993 ts_tablet_manager.cc:902] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:52.013193 9927 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:52.013927 13917 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.021230 12223 raft_consensus.cc:1203] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.025722 9796 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:52.026144 9667 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:52.026744 9535 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:14:52.027552 10255 consensus_peers.cc:485] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e11a6197ccdd4e8d9a203efae4a94988. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:52.027946 13926 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.028461 13926 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.028998 13926 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.033478 14032 ts_tablet_manager.cc:902] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:52.034355 12090 raft_consensus.cc:1203] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.057193 13993 tablet_copy_client.cc:308] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.6:37175
I20240503 08:14:52.057304 13917 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:52.057984 10211 tablet_copy_service.cc:140] P fa60e870de844325bad4b9185eaee468: Received BeginTabletCopySession request for tablet a61f59d188824bf3b610771b2fc05f06 from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:40727)
I20240503 08:14:52.058142 10211 tablet_copy_service.cc:161] P fa60e870de844325bad4b9185eaee468: Beginning new tablet copy session on tablet a61f59d188824bf3b610771b2fc05f06 from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:40727: session id = ca5fa388d5b84a0abac1c05275155dcf-a61f59d188824bf3b610771b2fc05f06
I20240503 08:14:52.061084 10211 tablet_copy_source_session.cc:205] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:52.062420 13993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61f59d188824bf3b610771b2fc05f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:52.062289 9926 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:52.068696 13993 tablet_copy_client.cc:783] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:52.068956 13993 tablet_copy_client.cc:647] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:52.069473 9665 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:52.074177 13937 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:52.074669 13937 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:52.074939 13937 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.365s user 0.077s sys 0.034s
I20240503 08:14:52.075749 13937 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:52.076057 13937 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:52.076282 13937 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:52.077085 13937 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:52.077832 9385 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:52.080564 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-1be499fcc2bf4716b16a5983950d3ed8 received from {username='slave'} at 127.24.123.9:55179
I20240503 08:14:52.080704 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-1be499fcc2bf4716b16a5983950d3ed8 on tablet 1be499fcc2bf4716b16a5983950d3ed8 with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:52.087563 12090 raft_consensus.cc:1203] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.087824 10058 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:52.090183 13993 tablet_copy_client.cc:523] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:52.094024 13993 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:52.096864 13998 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:52.097275 13998 tablet_bootstrap.cc:492] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:52.097538 13998 ts_tablet_manager.cc:1367] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.093s user 0.043s sys 0.007s
I20240503 08:14:52.098352 13998 raft_consensus.cc:348] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:52.098657 13998 raft_consensus.cc:729] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:52.098722 12090 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:52.099138 13998 consensus_queue.cc:260] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:52.099776 13998 ts_tablet_manager.cc:1398] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:14:52.100431 10211 tablet_copy_service.cc:342] P fa60e870de844325bad4b9185eaee468: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-cc9f0afdc01d4d03917b9040c2053426 received from {username='slave'} at 127.24.123.10:34373
I20240503 08:14:52.100564 10211 tablet_copy_service.cc:433] P fa60e870de844325bad4b9185eaee468: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-cc9f0afdc01d4d03917b9040c2053426 on tablet cc9f0afdc01d4d03917b9040c2053426 with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:52.121937 9797 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:52.134372 9534 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:52.141786 14035 tablet_copy_client.cc:308] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:52.142541 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 9428100e4a364805863419b22ca3c682 from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:50757)
I20240503 08:14:52.142714 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 9428100e4a364805863419b22ca3c682 from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:50757: session id = ca5fa388d5b84a0abac1c05275155dcf-9428100e4a364805863419b22ca3c682
I20240503 08:14:52.143916 14032 tablet_copy_client.cc:308] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:52.145524 9555 tablet_copy_source_session.cc:205] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:52.147922 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 8cb26d36a69d4f4ab43401822096756f from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:55179)
I20240503 08:14:52.148265 14035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:52.148591 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 8cb26d36a69d4f4ab43401822096756f from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:55179: session id = 3a70e21b51a142bd8f7dc93fa58742fb-8cb26d36a69d4f4ab43401822096756f
I20240503 08:14:52.152390 9555 tablet_copy_source_session.cc:205] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:52.153811 14032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb26d36a69d4f4ab43401822096756f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:52.157049 14035 tablet_copy_client.cc:783] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:52.157547 14035 tablet_copy_client.cc:647] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:52.159603 14035 tablet_copy_client.cc:523] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:52.160809 14032 tablet_copy_client.cc:783] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:52.161106 14032 tablet_copy_client.cc:647] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:52.161779 10058 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:52.162760 14032 tablet_copy_client.cc:523] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:52.163954 14035 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:52.166648 14032 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
W20240503 08:14:52.168411 9995 consensus_peers.cc:485] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 -> Peer 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439): Couldn't send request to peer 925c339b047b4cf5bae840c5a357d3de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43f3607501a84050962dc116210c09cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:52.169507 11958 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:52.170338 13872 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.175238 9796 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:52.175798 9665 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:52.175787 13858 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.176455 13858 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.182432 10189 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:52.184301 10319 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:52.185272 9534 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:52.185637 13872 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.186482 13872 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.196156 13998 ts_tablet_manager.cc:902] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:52.205699 13993 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:52.213002 14095 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:52.214087 14093 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:52.216192 9796 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:52.225425 9387 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:52.227679 13993 tablet_bootstrap.cc:492] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:52.227980 13993 ts_tablet_manager.cc:1367] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.134s user 0.065s sys 0.012s
I20240503 08:14:52.228739 13993 raft_consensus.cc:348] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:52.229060 13993 raft_consensus.cc:729] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:52.229295 13993 consensus_queue.cc:260] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:52.229884 13993 ts_tablet_manager.cc:1398] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:14:52.232626 10211 tablet_copy_service.cc:342] P fa60e870de844325bad4b9185eaee468: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-a61f59d188824bf3b610771b2fc05f06 received from {username='slave'} at 127.24.123.8:40727
I20240503 08:14:52.232800 10211 tablet_copy_service.cc:433] P fa60e870de844325bad4b9185eaee468: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-a61f59d188824bf3b610771b2fc05f06 on tablet a61f59d188824bf3b610771b2fc05f06 with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:52.236431 9665 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:52.243021 9534 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:52.245906 13998 tablet_copy_client.cc:308] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:41805
I20240503 08:14:52.246610 9818 tablet_copy_service.cc:140] P 8de08e679a904ee48ba659861c730725: Received BeginTabletCopySession request for tablet 2a2867608f0a4d69a3e169a59e358d10 from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:51097)
I20240503 08:14:52.246783 9818 tablet_copy_service.cc:161] P 8de08e679a904ee48ba659861c730725: Beginning new tablet copy session on tablet 2a2867608f0a4d69a3e169a59e358d10 from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:51097: session id = 925c339b047b4cf5bae840c5a357d3de-2a2867608f0a4d69a3e169a59e358d10
I20240503 08:14:52.249125 9818 tablet_copy_source_session.cc:205] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:52.252537 11958 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:52.253151 13998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2867608f0a4d69a3e169a59e358d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:52.259693 13998 tablet_copy_client.cc:783] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:52.260210 13998 tablet_copy_client.cc:647] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:52.260035 10319 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:52.263957 13998 tablet_copy_client.cc:523] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:52.265321 10189 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } } }
I20240503 08:14:52.267694 13998 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:52.285921 14035 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:52.290011 14035 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:52.290609 14035 ts_tablet_manager.cc:1367] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.127s user 0.056s sys 0.006s
I20240503 08:14:52.290746 11958 raft_consensus.cc:1203] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.292516 14032 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:52.292958 14032 tablet_bootstrap.cc:492] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:52.293210 14032 ts_tablet_manager.cc:1367] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.127s user 0.046s sys 0.004s
I20240503 08:14:52.294073 14032 raft_consensus.cc:348] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:52.294461 14032 raft_consensus.cc:729] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:52.294754 14032 consensus_queue.cc:260] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:52.295595 14032 ts_tablet_manager.cc:1398] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:52.297302 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-8cb26d36a69d4f4ab43401822096756f received from {username='slave'} at 127.24.123.9:55179
I20240503 08:14:52.297451 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-8cb26d36a69d4f4ab43401822096756f on tablet 8cb26d36a69d4f4ab43401822096756f with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:52.299552 14035 raft_consensus.cc:348] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:52.303321 14035 raft_consensus.cc:729] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:52.304033 14035 consensus_queue.cc:260] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:52.306586 14035 ts_tablet_manager.cc:1398] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:14:52.308656 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-9428100e4a364805863419b22ca3c682 received from {username='slave'} at 127.24.123.8:50757
I20240503 08:14:52.308806 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-9428100e4a364805863419b22ca3c682 on tablet 9428100e4a364805863419b22ca3c682 with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:52.342801 13716 raft_consensus.cc:1050] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:52.344003 13716 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:52.345633 13998 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:52.346055 13998 tablet_bootstrap.cc:492] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:52.346341 13998 ts_tablet_manager.cc:1367] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.079s user 0.041s sys 0.005s
I20240503 08:14:52.347182 13998 raft_consensus.cc:348] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:52.347492 13998 raft_consensus.cc:729] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:52.347702 13998 consensus_queue.cc:260] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:52.348358 13998 ts_tablet_manager.cc:1398] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:14:52.349608 9796 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.350425 13853 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.356205 12224 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.356611 9818 tablet_copy_service.cc:342] P 8de08e679a904ee48ba659861c730725: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-2a2867608f0a4d69a3e169a59e358d10 received from {username='slave'} at 127.24.123.10:51097
I20240503 08:14:52.356748 9818 tablet_copy_service.cc:433] P 8de08e679a904ee48ba659861c730725: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-2a2867608f0a4d69a3e169a59e358d10 on tablet 2a2867608f0a4d69a3e169a59e358d10 with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:52.359318 11958 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.360694 13863 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.361317 13863 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.384075 9665 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.384706 13853 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.396891 10189 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.396898 10058 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.397512 13863 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.397866 13853 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.398383 10319 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.399039 13853 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.404695 13863 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.408510 10319 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.408638 9665 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.409065 10058 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.412874 9387 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.418018 10189 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.419034 12224 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.418048 11958 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.435734 9796 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.442924 9534 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:52.446341 9665 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.446885 10058 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.447201 14139 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.447687 12224 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.447917 11958 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.448454 14139 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.449141 14123 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.449605 10319 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.449559 14123 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.450241 14139 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.450444 10189 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.450959 14139 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.452673 14139 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.453375 10322 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.458343 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fe97b80e1294c4791560daf991bfcdb with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:52.458997 11958 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.459161 10058 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.463284 12089 raft_consensus.cc:1203] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.464169 9667 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.468019 9390 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.473290 11957 raft_consensus.cc:1203] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.480607 10188 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.482088 9777 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe97b80e1294c4791560daf991bfcdb with delete_type TABLET_DATA_TOMBSTONED (TS 8de08e679a904ee48ba659861c730725 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51550
I20240503 08:14:52.485302 11957 raft_consensus.cc:1203] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.502679 14147 tablet_replica.cc:330] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725: stopping tablet replica
I20240503 08:14:52.500257 12224 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.505017 14147 raft_consensus.cc:2229] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:52.505422 14147 raft_consensus.cc:2258] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:52.507889 14147 ts_tablet_manager.cc:1875] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:52.514945 14147 ts_tablet_manager.cc:1888] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:52.515278 14147 log.cc:1196] T 3fe97b80e1294c4791560daf991bfcdb P 8de08e679a904ee48ba659861c730725: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/3fe97b80e1294c4791560daf991bfcdb
I20240503 08:14:52.516358 9378 catalog_manager.cc:4830] TS 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): tablet 3fe97b80e1294c4791560daf991bfcdb (table rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]) successfully deleted
I20240503 08:14:52.528685 13890 raft_consensus.cc:1050] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:52.529291 12224 raft_consensus.cc:1203] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.529831 13890 consensus_queue.cc:237] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:52.533354 12224 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.533957 14148 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.535956 10322 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.536018 11957 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.536306 10058 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.536536 9796 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.537042 14146 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.537351 14150 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.537904 14150 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.539425 13395 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.554358 9665 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.556325 14146 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.556740 9534 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.557294 14145 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.558564 13890 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.563514 10060 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.564657 11957 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.568548 10322 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.570310 9796 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.571063 9385 catalog_manager.cc:5484] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.578450 12224 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.579499 12089 raft_consensus.cc:1203] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.587149 12224 raft_consensus.cc:1203] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.590627 9665 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.597507 9534 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
W20240503 08:14:52.602634 13890 consensus_queue.cc:1159] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b3d3436263d04c57b2d47cc9d766efa5" responder_term: 1 status { last_received { term: 1 index: 9 } last_committed_idx: 9 last_received_current_leader { term: 1 index: 9 } }
I20240503 08:14:52.603021 14035 ts_tablet_manager.cc:902] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:52.603117 10188 consensus_queue.cc:237] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:52.606700 14035 tablet_copy_client.cc:308] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.7:42473
I20240503 08:14:52.607091 11957 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.607143 10322 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.607524 10342 tablet_copy_service.cc:140] P 0cb18b8b10f944e39829faf16e061fd6: Received BeginTabletCopySession request for tablet e11a6197ccdd4e8d9a203efae4a94988 from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:39157)
I20240503 08:14:52.607616 9534 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.607719 10342 tablet_copy_service.cc:161] P 0cb18b8b10f944e39829faf16e061fd6: Beginning new tablet copy session on tablet e11a6197ccdd4e8d9a203efae4a94988 from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:39157: session id = ca5fa388d5b84a0abac1c05275155dcf-e11a6197ccdd4e8d9a203efae4a94988
I20240503 08:14:52.608243 9796 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.608395 10060 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.608520 13890 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.608891 14150 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.609261 12224 raft_consensus.cc:1261] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.609382 14148 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.609707 14146 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.609956 13890 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.610512 10342 tablet_copy_source_session.cc:205] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:52.614387 13890 consensus_queue.cc:1035] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.615351 14146 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.616708 14035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:52.616300 9797 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.618939 10322 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.618824 11957 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.619015 12223 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.627398 13917 raft_consensus.cc:1050] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:52.627820 9389 catalog_manager.cc:5484] T 014f053652224d60ac8ef85064f0cabe P ca5fa388d5b84a0abac1c05275155dcf reported cstate change: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.628614 13917 consensus_queue.cc:237] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:52.629338 14035 tablet_copy_client.cc:783] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:52.629561 14035 tablet_copy_client.cc:647] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:52.633111 14035 tablet_copy_client.cc:523] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:52.633971 13858 raft_consensus.cc:1050] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:52.635183 13858 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:52.635138 10060 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.636654 14035 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:52.643788 13998 ts_tablet_manager.cc:902] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: Initiating tablet copy from peer a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:52.644589 9379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 014f053652224d60ac8ef85064f0cabe with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:52.646957 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 014f053652224d60ac8ef85064f0cabe with delete_type TABLET_DATA_TOMBSTONED (TS d1eb5e69c6264a20a6e8297cc5deb8d1 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55050
I20240503 08:14:52.647706 9534 raft_consensus.cc:2932] T 014f053652224d60ac8ef85064f0cabe P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:52.655200 14176 tablet_replica.cc:330] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: stopping tablet replica
I20240503 08:14:52.656045 14176 raft_consensus.cc:2229] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:52.657176 14176 raft_consensus.cc:2258] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:52.658077 12089 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.658691 13919 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.660714 14176 ts_tablet_manager.cc:1875] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:52.665117 9797 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.665146 9665 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.665524 9534 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.665709 13919 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.666074 12223 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.666230 13926 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.666337 9927 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.666563 13919 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.667021 13919 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20240503 08:14:52.673295 13926 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.677032 10060 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.677887 13926 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.710042 14176 ts_tablet_manager.cc:1888] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:52.711318 14176 log.cc:1196] T 014f053652224d60ac8ef85064f0cabe P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/014f053652224d60ac8ef85064f0cabe
I20240503 08:14:52.712419 9380 catalog_manager.cc:4830] TS d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123): tablet 014f053652224d60ac8ef85064f0cabe (table rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]) successfully deleted
I20240503 08:14:52.714231 12089 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.715041 14095 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.719290 12224 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.719511 10319 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.720104 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.721305 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.731547 9797 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.732371 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.744663 9667 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.745450 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.757776 9535 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.758534 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.764667 14035 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:52.765189 14035 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:52.765499 14035 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.129s user 0.052s sys 0.002s
I20240503 08:14:52.766333 14035 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:52.766655 14035 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:52.766943 14035 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:52.767850 13926 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.769366 9927 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.769968 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.770448 14035 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:14:52.772998 9665 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.775424 10342 tablet_copy_service.cc:342] P 0cb18b8b10f944e39829faf16e061fd6: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-e11a6197ccdd4e8d9a203efae4a94988 received from {username='slave'} at 127.24.123.8:39157
I20240503 08:14:52.775557 10342 tablet_copy_service.cc:433] P 0cb18b8b10f944e39829faf16e061fd6: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-e11a6197ccdd4e8d9a203efae4a94988 on tablet e11a6197ccdd4e8d9a203efae4a94988 with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:52.776592 12223 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.782487 9390 catalog_manager.cc:5484] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf reported cstate change: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.788733 9796 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.796463 12089 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.796880 9927 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.797905 10058 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.804838 9534 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.812481 10322 consensus_queue.cc:237] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:52.815575 12089 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.816144 9665 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.816273 12223 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.817134 13926 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.817734 13926 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.819484 9796 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.820197 14184 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.821527 14184 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.824247 10058 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.825232 14184 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.826205 13998 tablet_copy_client.cc:308] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43643
I20240503 08:14:52.826373 14184 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.827172 10080 tablet_copy_service.cc:140] P a1402a75453d4ccf992606dcce77dc51: Received BeginTabletCopySession request for tablet 43f3607501a84050962dc116210c09cd from peer 925c339b047b4cf5bae840c5a357d3de ({username='slave'} at 127.24.123.10:44419)
I20240503 08:14:52.827333 10080 tablet_copy_service.cc:161] P a1402a75453d4ccf992606dcce77dc51: Beginning new tablet copy session on tablet 43f3607501a84050962dc116210c09cd from peer 925c339b047b4cf5bae840c5a357d3de at {username='slave'} at 127.24.123.10:44419: session id = 925c339b047b4cf5bae840c5a357d3de-43f3607501a84050962dc116210c09cd
I20240503 08:14:52.830199 10080 tablet_copy_source_session.cc:205] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:52.831916 13998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:52.835790 12224 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.836550 9665 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.839769 10058 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.844638 13998 tablet_copy_client.cc:783] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:52.844895 13998 tablet_copy_client.cc:647] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:52.844066 14108 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.847318 13998 tablet_copy_client.cc:523] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:52.848284 9667 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.849498 12089 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.852072 13998 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: Bootstrap starting.
I20240503 08:14:52.851969 9797 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.852556 9385 catalog_manager.cc:5484] T fdb88bebfdab4e54abc5dba5834a3298 P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.855885 9387 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de reported cstate change: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.857012 12224 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.857101 9534 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.858310 10322 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.860033 9929 raft_consensus.cc:1261] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.860849 14180 consensus_queue.cc:1035] T fdb88bebfdab4e54abc5dba5834a3298 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.861680 9513 tablet_service.cc:1514] Processing DeleteTablet for tablet fdb88bebfdab4e54abc5dba5834a3298 with delete_type TABLET_DATA_TOMBSTONED (TS b3d3436263d04c57b2d47cc9d766efa5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55994
I20240503 08:14:52.863627 9929 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.867921 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fdb88bebfdab4e54abc5dba5834a3298 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:52.869246 12089 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.872464 9798 raft_consensus.cc:2932] T fdb88bebfdab4e54abc5dba5834a3298 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.890520 10060 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:52.898802 12089 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.899392 14095 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.899935 12224 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.900701 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.902045 10322 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.902923 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.903050 14139 raft_consensus.cc:1050] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:52.903434 9928 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.904096 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.904141 14139 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:52.905396 9928 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.915225 9798 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.915856 14108 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.917906 9667 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:52.918610 13858 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:52.919689 14108 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.923190 9667 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.925253 11957 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.926010 12089 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.926705 10059 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.926843 10322 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.927062 14137 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.927548 9925 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.927518 13853 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.927749 13716 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.928143 13863 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.929239 13853 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.932354 9664 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.932546 9797 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:52.933187 14123 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.933674 14123 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:52.924908 9798 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.936604 14146 raft_consensus.cc:1050] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:52.937670 14146 consensus_queue.cc:237] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:52.944025 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f6758eb87b047d29a060fcbfb775903 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:52.945433 12090 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.945605 10321 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.947225 12224 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.954342 9389 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.973071 9514 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6758eb87b047d29a060fcbfb775903 with delete_type TABLET_DATA_TOMBSTONED (TS b3d3436263d04c57b2d47cc9d766efa5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55994
I20240503 08:14:52.980182 13998 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:52.980777 13998 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: Bootstrap complete.
I20240503 08:14:52.981231 13998 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: Time spent bootstrapping tablet: real 0.129s user 0.045s sys 0.014s
I20240503 08:14:52.980613 10060 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.981906 14123 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.982491 13998 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:52.982784 13998 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925c339b047b4cf5bae840c5a357d3de, State: Initialized, Role: LEARNER
I20240503 08:14:52.983059 13998 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: NON_VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: true } }
I20240503 08:14:52.983703 13998 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:52.986420 10080 tablet_copy_service.cc:342] P a1402a75453d4ccf992606dcce77dc51: Request end of tablet copy session 925c339b047b4cf5bae840c5a357d3de-43f3607501a84050962dc116210c09cd received from {username='slave'} at 127.24.123.10:44419
I20240503 08:14:52.986543 10080 tablet_copy_service.cc:433] P a1402a75453d4ccf992606dcce77dc51: ending tablet copy session 925c339b047b4cf5bae840c5a357d3de-43f3607501a84050962dc116210c09cd on tablet 43f3607501a84050962dc116210c09cd with peer 925c339b047b4cf5bae840c5a357d3de
I20240503 08:14:52.988178 12224 raft_consensus.cc:1203] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:52.990139 12089 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.992117 11957 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.992647 9385 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.993283 9665 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.992429 9926 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:52.998488 10319 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:52.986167 9797 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.002528 14202 tablet_replica.cc:330] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5: stopping tablet replica
I20240503 08:14:53.003053 14202 raft_consensus.cc:2229] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.003427 14202 raft_consensus.cc:2258] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.005522 9928 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.017366 9535 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:53.022203 14202 ts_tablet_manager.cc:1875] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.023315 11957 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.023836 10060 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.024194 12089 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.025344 9928 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.023916 14139 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.027841 13863 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.028124 14123 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.028613 13853 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.029179 10319 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.029752 9667 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.031925 13863 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240503 08:14:53.032734 14215 raft_consensus.cc:1050] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:53.032815 13853 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.033803 14215 consensus_queue.cc:237] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:53.034664 14139 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.036343 10319 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.041908 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1be499fcc2bf4716b16a5983950d3ed8 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:53.042095 14202 ts_tablet_manager.cc:1888] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.042299 14202 log.cc:1196] T fdb88bebfdab4e54abc5dba5834a3298 P b3d3436263d04c57b2d47cc9d766efa5: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/fdb88bebfdab4e54abc5dba5834a3298
I20240503 08:14:53.042697 11957 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.042850 12090 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.043434 9378 catalog_manager.cc:4830] TS b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353): tablet fdb88bebfdab4e54abc5dba5834a3298 (table rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]) successfully deleted
I20240503 08:14:53.043387 14202 tablet_replica.cc:330] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5: stopping tablet replica
I20240503 08:14:53.043846 14202 raft_consensus.cc:2229] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.044157 14202 raft_consensus.cc:2258] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.045886 14202 ts_tablet_manager.cc:1875] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.046124 9390 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.049651 10060 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.053186 9777 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 with delete_type TABLET_DATA_TOMBSTONED (TS 8de08e679a904ee48ba659861c730725 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51550
I20240503 08:14:53.055750 14150 raft_consensus.cc:1050] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:53.055974 9928 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.056762 14150 consensus_queue.cc:237] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:53.058813 14202 ts_tablet_manager.cc:1888] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.060326 14202 log.cc:1196] T 0f6758eb87b047d29a060fcbfb775903 P b3d3436263d04c57b2d47cc9d766efa5: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f6758eb87b047d29a060fcbfb775903
I20240503 08:14:53.062592 9378 catalog_manager.cc:4830] TS b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353): tablet 0f6758eb87b047d29a060fcbfb775903 (table rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]) successfully deleted
I20240503 08:14:53.062772 10189 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.063529 14217 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.063947 11957 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.064378 9797 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.065097 14206 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.067417 12224 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.068188 14148 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.070472 12090 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.070979 14148 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.072276 9535 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.073314 14217 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.076360 13853 raft_consensus.cc:1050] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:53.077587 13853 consensus_queue.cc:237] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:53.079890 9665 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.079905 9797 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.080888 9533 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.081666 14145 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.082329 14145 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.082757 14145 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.083302 14216 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.085691 12089 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.087546 14216 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.103677 10319 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.104135 12088 raft_consensus.cc:1203] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:53.104450 14148 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.079965 10060 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.110066 14148 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.112354 12088 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.112824 10322 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.113009 9795 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.113080 11958 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.113626 13395 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.114190 14219 tablet_replica.cc:330] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: stopping tablet replica
I20240503 08:14:53.114121 13395 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.114384 14219 raft_consensus.cc:2229] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.114511 10059 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.114589 13890 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.114672 14219 raft_consensus.cc:2258] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.115238 13890 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.115674 14219 ts_tablet_manager.cc:1875] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.122692 9664 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.042093 9667 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.124421 10322 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.124800 10057 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.125034 14221 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.127348 14145 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.127857 14145 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.127955 9534 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.128613 14145 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.130733 14216 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.134368 14217 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.139546 13800 raft_consensus.cc:1050] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:53.139850 10188 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.140779 13800 consensus_queue.cc:237] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:53.146142 9532 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.148921 9798 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.151849 14219 ts_tablet_manager.cc:1888] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.153033 9389 catalog_manager.cc:5484] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:53.153030 12089 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.170512 14123 raft_consensus.cc:1050] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:53.170818 14219 log.cc:1196] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/1be499fcc2bf4716b16a5983950d3ed8
I20240503 08:14:53.171473 14123 consensus_queue.cc:237] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:53.173816 9378 catalog_manager.cc:4830] TS 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): tablet 1be499fcc2bf4716b16a5983950d3ed8 (table rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]) successfully deleted
I20240503 08:14:53.175954 12087 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.176544 10320 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.176589 9798 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.176971 12223 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.177417 13863 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.178090 10056 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.178041 13863 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.181495 13863 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.181825 13716 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.187294 11957 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.189704 9928 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.191108 9667 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.192090 12086 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.192729 14017 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.200413 12224 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.200942 14017 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.208210 10321 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.211740 10189 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.211316 14145 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.212230 14017 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.217579 9798 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.218847 12224 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.219856 14123 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20240503 08:14:53.220738 9389 catalog_manager.cc:5484] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:53.221690 13716 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.222398 14137 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.226908 10322 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.227092 10056 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.227924 14017 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.229511 14017 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.232077 10055 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.231760 12090 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.232795 9666 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.234102 10189 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.234948 14139 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.235905 14139 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.236518 14139 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.227439 9665 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.248544 11956 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.249035 9797 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.249748 13863 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.251053 13863 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.252089 9533 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.260443 10321 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.265851 14123 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.271163 10320 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.273031 12089 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.273646 14123 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.278730 13890 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.279624 12088 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.282598 10058 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.284845 10320 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.293684 9389 catalog_manager.cc:5484] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.293901 13863 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.297663 9533 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
W20240503 08:14:53.300248 9732 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.6:37175, user_credentials={real_user=slave}} blocked reactor thread for 55249.8us
I20240503 08:14:53.302428 10188 consensus_queue.cc:237] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:53.306083 12085 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.306720 14145 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.307719 9665 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.308853 9798 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.310443 14017 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.311466 12090 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.315929 9534 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.316555 9795 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.316494 14262 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.317272 14145 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.317162 12222 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.306857 10319 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.318666 14150 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.319756 14260 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.322343 10189 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.324055 9534 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.327924 9389 catalog_manager.cc:5484] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.328141 9392 catalog_manager.cc:5484] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:53.329176 9795 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.330822 12088 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.328440 13863 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.339345 10191 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.342072 14146 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.343430 10319 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.347407 12223 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.348378 14150 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.350070 12224 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.350713 9385 catalog_manager.cc:5484] T a61f59d188824bf3b610771b2fc05f06 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.354238 9795 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.354668 12221 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.355904 11957 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.361409 10055 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.363885 10056 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.364440 14150 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.368081 10056 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.369273 9379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a61f59d188824bf3b610771b2fc05f06 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:53.369711 9389 catalog_manager.cc:5484] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.369828 9667 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.375625 12088 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.376571 10322 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.379725 10058 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.381038 10060 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.383584 10054 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.388485 10319 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
W20240503 08:14:53.391775 9471 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.4:38565, user_credentials={real_user=slave}} blocked reactor thread for 182159us
I20240503 08:14:53.393756 9795 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.397030 10321 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.401285 12089 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.407332 9928 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.409361 11958 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.411583 9533 raft_consensus.cc:1261] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.412498 14146 consensus_queue.cc:1035] T a61f59d188824bf3b610771b2fc05f06 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.423127 9533 raft_consensus.cc:2932] T a61f59d188824bf3b610771b2fc05f06 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.423787 9534 consensus_queue.cc:237] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:53.427230 10188 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.427852 10321 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.427796 13716 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.428268 11957 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.432180 10056 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.433430 13853 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20240503 08:14:53.433782 13863 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.435060 13863 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.435554 12088 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.436606 13863 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.444451 13853 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.450982 10321 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.457041 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet a61f59d188824bf3b610771b2fc05f06 with delete_type TABLET_DATA_TOMBSTONED (TS d1eb5e69c6264a20a6e8297cc5deb8d1 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55050
I20240503 08:14:53.456951 9388 catalog_manager.cc:5484] T 8cb26d36a69d4f4ab43401822096756f P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.459187 11958 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.459370 10188 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.461477 9777 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cb26d36a69d4f4ab43401822096756f with delete_type TABLET_DATA_TOMBSTONED (TS 8de08e679a904ee48ba659861c730725 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51550
I20240503 08:14:53.461877 14219 tablet_replica.cc:330] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725: stopping tablet replica
I20240503 08:14:53.462082 14219 raft_consensus.cc:2229] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.462347 14219 raft_consensus.cc:2258] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.463357 14219 ts_tablet_manager.cc:1875] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.466418 10057 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.467675 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cb26d36a69d4f4ab43401822096756f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:53.469147 12090 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.469815 14219 ts_tablet_manager.cc:1888] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.469926 14219 log.cc:1196] T 8cb26d36a69d4f4ab43401822096756f P 8de08e679a904ee48ba659861c730725: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/8cb26d36a69d4f4ab43401822096756f
I20240503 08:14:53.469514 10056 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.470767 9378 catalog_manager.cc:4830] TS 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): tablet 8cb26d36a69d4f4ab43401822096756f (table rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]) successfully deleted
I20240503 08:14:53.471051 9663 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.471279 9665 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.477257 9666 consensus_queue.cc:237] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:53.484081 9662 raft_consensus.cc:1261] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.487972 9533 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.488329 13863 consensus_queue.cc:1035] T 8cb26d36a69d4f4ab43401822096756f P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.490300 10188 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.491114 12090 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.491408 11958 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.491621 14215 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240503 08:14:53.492512 9661 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.494271 10321 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.493734 14215 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240503 08:14:53.495307 14205 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240503 08:14:53.495810 14221 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.499526 14217 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20240503 08:14:53.504683 10188 consensus_queue.cc:237] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:53.506698 9667 raft_consensus.cc:2932] T 8cb26d36a69d4f4ab43401822096756f P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.506824 14221 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.508983 12089 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.508605 12088 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.510038 10319 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.509814 10191 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.510571 14145 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.511036 14145 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.511140 14278 tablet_replica.cc:330] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1: stopping tablet replica
I20240503 08:14:53.511327 14278 raft_consensus.cc:2229] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.511570 14278 raft_consensus.cc:2258] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.513075 14278 ts_tablet_manager.cc:1875] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.515902 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8506211351284b7a8e2a166b8083977a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:53.516049 9385 catalog_manager.cc:5484] T 8506211351284b7a8e2a166b8083977a P 925c339b047b4cf5bae840c5a357d3de reported cstate change: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New cstate: current_term: 1 leader_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.517148 11957 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.518704 9533 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.521235 9777 tablet_service.cc:1514] Processing DeleteTablet for tablet 8506211351284b7a8e2a166b8083977a with delete_type TABLET_DATA_TOMBSTONED (TS 8de08e679a904ee48ba659861c730725 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51550
I20240503 08:14:53.521378 10056 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.521994 14148 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.523065 14219 tablet_replica.cc:330] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725: stopping tablet replica
I20240503 08:14:53.523339 14219 raft_consensus.cc:2229] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.524416 14219 raft_consensus.cc:2258] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.526626 9795 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.527493 14148 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.528549 14219 ts_tablet_manager.cc:1875] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.531540 14095 raft_consensus.cc:1050] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: attempting to promote NON_VOTER 925c339b047b4cf5bae840c5a357d3de to VOTER
I20240503 08:14:53.531510 14148 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.531908 9795 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.532722 14095 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:53.536908 9379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc9f0afdc01d4d03917b9040c2053426 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:53.538055 11958 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.538686 12090 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.539625 9385 catalog_manager.cc:5484] T cc9f0afdc01d4d03917b9040c2053426 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.543488 10059 raft_consensus.cc:1261] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer d1eb5e69c6264a20a6e8297cc5deb8d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.544457 13395 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.545488 10319 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.546532 9513 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9f0afdc01d4d03917b9040c2053426 with delete_type TABLET_DATA_TOMBSTONED (TS b3d3436263d04c57b2d47cc9d766efa5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55994
I20240503 08:14:53.546913 14202 tablet_replica.cc:330] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5: stopping tablet replica
I20240503 08:14:53.547119 14202 raft_consensus.cc:2229] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.547461 14202 raft_consensus.cc:2258] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.548381 14202 ts_tablet_manager.cc:1875] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.550810 10056 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.551748 11958 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.554590 14202 ts_tablet_manager.cc:1888] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.554769 14202 log.cc:1196] T cc9f0afdc01d4d03917b9040c2053426 P b3d3436263d04c57b2d47cc9d766efa5: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/cc9f0afdc01d4d03917b9040c2053426
I20240503 08:14:53.555634 9378 catalog_manager.cc:4830] TS b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353): tablet cc9f0afdc01d4d03917b9040c2053426 (table rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]) successfully deleted
I20240503 08:14:53.556748 10321 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.562705 12090 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 LEARNER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.563321 13858 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.567231 9795 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.567710 9533 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.521587 9663 raft_consensus.cc:1261] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.568105 14095 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.545572 14205 consensus_queue.cc:1035] T 8506211351284b7a8e2a166b8083977a P d1eb5e69c6264a20a6e8297cc5deb8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.568622 14095 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.568709 13395 consensus_queue.cc:1035] T cc9f0afdc01d4d03917b9040c2053426 P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.570387 9663 raft_consensus.cc:2932] T cc9f0afdc01d4d03917b9040c2053426 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.572610 10321 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.573132 13858 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.576049 10189 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.576607 13858 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.581424 11958 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.583503 10056 raft_consensus.cc:2932] T 8506211351284b7a8e2a166b8083977a P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:53.586835 13858 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.594398 14219 ts_tablet_manager.cc:1888] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.595619 14219 log.cc:1196] T 8506211351284b7a8e2a166b8083977a P 8de08e679a904ee48ba659861c730725: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/8506211351284b7a8e2a166b8083977a
I20240503 08:14:53.596644 9378 catalog_manager.cc:4830] TS 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): tablet 8506211351284b7a8e2a166b8083977a (table rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]) successfully deleted
I20240503 08:14:53.608948 13858 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.611932 14184 raft_consensus.cc:1050] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:14:53.612784 14278 ts_tablet_manager.cc:1888] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.613131 14184 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:53.613386 14278 log.cc:1196] T a61f59d188824bf3b610771b2fc05f06 P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/a61f59d188824bf3b610771b2fc05f06
I20240503 08:14:53.615221 9380 catalog_manager.cc:4830] TS d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123): tablet a61f59d188824bf3b610771b2fc05f06 (table rebalance_test_table_4 [id=604bdb89be354dd0936c90a5e32bffa6]) successfully deleted
I20240503 08:14:53.615106 10189 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.615712 10319 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.618398 9533 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.621129 12089 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.623198 9388 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.629832 11957 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.637049 9798 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.646905 12222 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.647534 12089 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.647857 9661 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.648253 9928 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.648950 14197 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.649499 14197 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.649919 14197 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.656327 14095 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.656513 10056 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.657402 14180 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.662124 9795 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.662124 9533 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.662799 14180 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.663224 9661 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:53.663347 14180 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.663735 14197 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:53.673884 9667 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 925c339b047b4cf5bae840c5a357d3de (127.24.123.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.674453 14197 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.680312 9795 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.686432 9388 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.703892 9928 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.704257 12089 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.717672 9667 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.718859 10057 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.720626 12222 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.721923 9533 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.732580 10319 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:53.735965 12089 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.736258 10057 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.736698 9928 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.736837 14197 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.737345 14184 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.737640 12222 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.737874 14197 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.738454 9795 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.739359 9534 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.740674 14197 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.740556 14296 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.740972 14301 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.741217 10056 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.742146 14184 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.743309 12222 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.747674 9795 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.752880 9388 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.751889 9533 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.757534 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet e11a6197ccdd4e8d9a203efae4a94988 with delete_type TABLET_DATA_TOMBSTONED (TS d1eb5e69c6264a20a6e8297cc5deb8d1 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55050
I20240503 08:14:53.757943 14278 tablet_replica.cc:330] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1: stopping tablet replica
I20240503 08:14:53.758136 14278 raft_consensus.cc:2229] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.758419 14278 raft_consensus.cc:2258] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.760326 14278 ts_tablet_manager.cc:1875] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.761469 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e11a6197ccdd4e8d9a203efae4a94988 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:53.768041 12089 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.768663 9928 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.776388 14278 ts_tablet_manager.cc:1888] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.776531 14278 log.cc:1196] T e11a6197ccdd4e8d9a203efae4a94988 P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/e11a6197ccdd4e8d9a203efae4a94988
I20240503 08:14:53.777367 9380 catalog_manager.cc:4830] TS d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123): tablet e11a6197ccdd4e8d9a203efae4a94988 (table rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]) successfully deleted
I20240503 08:14:53.812888 9533 consensus_queue.cc:237] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:14:53.816113 12089 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.815939 12222 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.817080 9928 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.817586 13863 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.817857 9795 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.818416 14123 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.818737 10319 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.820582 10056 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.820605 13853 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.821336 13863 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.821928 13863 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.824251 14123 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.824810 14123 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.825402 9928 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.826479 10056 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.831483 9388 catalog_manager.cc:5484] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.833091 12089 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.835780 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 9428100e4a364805863419b22ca3c682 with delete_type TABLET_DATA_TOMBSTONED (TS d1eb5e69c6264a20a6e8297cc5deb8d1 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55050
I20240503 08:14:53.836192 14278 tablet_replica.cc:330] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1: stopping tablet replica
I20240503 08:14:53.836390 14278 raft_consensus.cc:2229] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.836627 14278 raft_consensus.cc:2258] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.838196 14278 ts_tablet_manager.cc:1875] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.838490 10319 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.843369 14278 ts_tablet_manager.cc:1888] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.843530 14278 log.cc:1196] T 9428100e4a364805863419b22ca3c682 P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/9428100e4a364805863419b22ca3c682
I20240503 08:14:53.844312 9380 catalog_manager.cc:4830] TS d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123): tablet 9428100e4a364805863419b22ca3c682 (table rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]) successfully deleted
I20240503 08:14:53.852988 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9428100e4a364805863419b22ca3c682 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:53.854308 9795 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.855069 12222 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:14:53.907212 9795 consensus_queue.cc:237] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:53.910861 12089 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.911046 10056 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.911177 12222 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.911473 14266 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.911669 10189 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.911693 9667 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.912323 14343 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240503 08:14:53.912849 10319 raft_consensus.cc:1261] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:53.913427 14266 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.913717 14342 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.913954 14262 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.914207 14256 consensus_queue.cc:1035] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:53.917692 14266 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.918262 10319 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.922128 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2a2867608f0a4d69a3e169a59e358d10 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:53.922713 10189 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.924286 9392 catalog_manager.cc:5484] T 2a2867608f0a4d69a3e169a59e358d10 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.928277 12222 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.935281 9513 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2867608f0a4d69a3e169a59e358d10 with delete_type TABLET_DATA_TOMBSTONED (TS b3d3436263d04c57b2d47cc9d766efa5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55994
I20240503 08:14:53.935670 14202 tablet_replica.cc:330] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5: stopping tablet replica
I20240503 08:14:53.935854 14202 raft_consensus.cc:2229] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:53.936100 14202 raft_consensus.cc:2258] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:53.937857 14202 ts_tablet_manager.cc:1875] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:53.937417 10056 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.941610 9667 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:53.945513 14202 ts_tablet_manager.cc:1888] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:53.945739 14202 log.cc:1196] T 2a2867608f0a4d69a3e169a59e358d10 P b3d3436263d04c57b2d47cc9d766efa5: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/2a2867608f0a4d69a3e169a59e358d10
I20240503 08:14:53.947372 9378 catalog_manager.cc:4830] TS b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353): tablet 2a2867608f0a4d69a3e169a59e358d10 (table rebalance_test_table_0 [id=8c572ec3b3b54b5fa75f76140ea1ef18]) successfully deleted
I20240503 08:14:54.150797 10056 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } }
I20240503 08:14:54.153654 11957 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:54.153739 12089 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:54.153980 10321 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:54.154557 10189 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:54.154605 9667 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:54.154709 9795 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:14:54.154860 14387 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:54.155241 14095 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:54.155565 14108 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:54.156018 14095 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:54.156512 14387 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:54.156742 13858 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:54.159662 13858 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:54.160235 11957 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:54.160653 9667 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:54.160657 9795 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:54.164618 10321 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:54.166795 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43f3607501a84050962dc116210c09cd with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:54.167989 9390 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:54.168133 12089 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:54.171479 10189 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:54.173219 9513 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f3607501a84050962dc116210c09cd with delete_type TABLET_DATA_TOMBSTONED (TS b3d3436263d04c57b2d47cc9d766efa5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55994
I20240503 08:14:54.173595 14202 tablet_replica.cc:330] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: stopping tablet replica
I20240503 08:14:54.173822 14202 raft_consensus.cc:2229] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:54.174154 14202 raft_consensus.cc:2258] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:54.175683 14202 ts_tablet_manager.cc:1875] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:54.180704 14202 ts_tablet_manager.cc:1888] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:14:54.180820 14202 log.cc:1196] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-0/wal/wals/43f3607501a84050962dc116210c09cd
I20240503 08:14:54.181449 9378 catalog_manager.cc:4830] TS b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353): tablet 43f3607501a84050962dc116210c09cd (table rebalance_test_table_5 [id=8435ea0a58044d31980daccba953a1e6]) successfully deleted
I20240503 08:14:54.301770 12089 raft_consensus.cc:2932] T 2a2867608f0a4d69a3e169a59e358d10 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } }
I20240503 08:14:54.381455 9909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:54.403285 9777 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:54.422384 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:14:54.439167 10038 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:54.452450 10169 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:54.475353 11938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.483592 9646 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:54.503795 12070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.514343 9513 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:54.545881 12204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:54.659399 9795 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:54.662544 12089 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:54.663127 10321 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:54.663947 14342 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:54.664256 10056 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:54.665834 14342 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:54.666597 14266 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:54.670670 9731 consensus_peers.cc:485] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49a60d2ed0744f35be8054955dcf8aa0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:54.675699 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:54.676008 9667 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:54.676072 9533 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:54.676313 14342 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:54.676842 14342 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:54.677182 14444 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:54.683336 14342 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:54.683876 9533 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:54.685616 10057 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:54.688771 12089 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:54.693879 9385 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:54.694185 10321 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:54.700289 10189 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:54.701793 9667 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:55.161098 14472 ts_tablet_manager.cc:902] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:55.162350 14472 tablet_copy_client.cc:308] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:41805
I20240503 08:14:55.168259 9818 tablet_copy_service.cc:140] P 8de08e679a904ee48ba659861c730725: Received BeginTabletCopySession request for tablet 49a60d2ed0744f35be8054955dcf8aa0 from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:42981)
I20240503 08:14:55.168442 9818 tablet_copy_service.cc:161] P 8de08e679a904ee48ba659861c730725: Beginning new tablet copy session on tablet 49a60d2ed0744f35be8054955dcf8aa0 from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:42981: session id = 3a70e21b51a142bd8f7dc93fa58742fb-49a60d2ed0744f35be8054955dcf8aa0
I20240503 08:14:55.170703 9818 tablet_copy_source_session.cc:205] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:55.171959 14472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:55.176551 14472 tablet_copy_client.cc:783] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:55.176784 14472 tablet_copy_client.cc:647] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:55.178243 14472 tablet_copy_client.cc:523] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:55.181267 14472 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:55.217382 14472 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:55.217814 14472 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:55.218111 14472 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.037s user 0.035s sys 0.000s
I20240503 08:14:55.218763 14472 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:55.219075 14472 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:55.219300 14472 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:55.220188 14472 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 08:14:55.220877 9818 tablet_copy_service.cc:342] P 8de08e679a904ee48ba659861c730725: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-49a60d2ed0744f35be8054955dcf8aa0 received from {username='slave'} at 127.24.123.9:42981
I20240503 08:14:55.221006 9818 tablet_copy_service.cc:433] P 8de08e679a904ee48ba659861c730725: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-49a60d2ed0744f35be8054955dcf8aa0 on tablet 49a60d2ed0744f35be8054955dcf8aa0 with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:55.643229 11957 raft_consensus.cc:1203] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:56.161144 14451 raft_consensus.cc:1050] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:14:56.162047 14451 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:56.165431 12090 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:56.165413 10321 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:56.165444 11957 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:56.166069 14491 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:56.167017 14266 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:56.167575 14266 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:56.174468 10057 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:56.174482 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:56.175136 14505 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:56.175386 14491 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:56.187235 14266 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:56.189451 10189 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:56.191418 12089 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:56.200377 9390 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:56.230846 9533 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:56.232748 10056 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:56.233502 14507 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:56.234361 10319 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:56.242302 9667 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:14:56.244470 11957 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:56.248548 14491 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:14:56.262168 9533 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:56.264688 9798 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0"
dest_uuid: "8de08e679a904ee48ba659861c730725"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45290
I20240503 08:14:56.264976 9798 raft_consensus.cc:595] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:14:56.270259 9667 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:56.271937 14507 raft_consensus.cc:979] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: : Instructing follower 3a70e21b51a142bd8f7dc93fa58742fb to start an election
I20240503 08:14:56.272114 14507 raft_consensus.cc:1067] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Signalling peer 3a70e21b51a142bd8f7dc93fa58742fb to start an election
I20240503 08:14:56.272961 11957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0"
dest_uuid: "3a70e21b51a142bd8f7dc93fa58742fb"
from {username='slave'} at 127.24.123.3:33553
I20240503 08:14:56.273263 11957 raft_consensus.cc:482] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:14:56.273445 11957 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:56.275954 11957 raft_consensus.cc:504] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:56.278306 11957 leader_election.cc:288] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [CANDIDATE]: Term 2 election: Requested vote from peers d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123), b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353), 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805), fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175), a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643), 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473), 925c339b047b4cf5bae840c5a357d3de (127.24.123.10:33439)
I20240503 08:14:56.278331 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "fa60e870de844325bad4b9185eaee468"
I20240503 08:14:56.278411 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "8de08e679a904ee48ba659861c730725"
I20240503 08:14:56.278574 10189 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:56.278604 9798 raft_consensus.cc:3032] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:14:56.278540 10056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a1402a75453d4ccf992606dcce77dc51"
I20240503 08:14:56.278697 9798 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Running, Role: LEADER
I20240503 08:14:56.278746 10056 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:56.279052 9798 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:56.279506 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1"
I20240503 08:14:56.279713 9667 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:56.279771 9798 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:56.281252 10189 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a70e21b51a142bd8f7dc93fa58742fb in term 2.
I20240503 08:14:56.281340 10056 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a70e21b51a142bd8f7dc93fa58742fb in term 2.
I20240503 08:14:56.282310 9667 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a70e21b51a142bd8f7dc93fa58742fb in term 2.
I20240503 08:14:56.283476 9798 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a70e21b51a142bd8f7dc93fa58742fb in term 2.
I20240503 08:14:56.288358 11893 leader_election.cc:302] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 8 voters: 5 yes votes; 0 no votes. yes voters: 3a70e21b51a142bd8f7dc93fa58742fb, 8de08e679a904ee48ba659861c730725, a1402a75453d4ccf992606dcce77dc51, d1eb5e69c6264a20a6e8297cc5deb8d1, fa60e870de844325bad4b9185eaee468; no voters:
I20240503 08:14:56.288808 14531 raft_consensus.cc:2781] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:56.289163 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
I20240503 08:14:56.289398 9533 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:56.295234 9533 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a70e21b51a142bd8f7dc93fa58742fb in term 2.
I20240503 08:14:56.305685 10319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "0cb18b8b10f944e39829faf16e061fd6"
I20240503 08:14:56.305976 10319 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:56.309772 10319 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a70e21b51a142bd8f7dc93fa58742fb in term 2.
I20240503 08:14:56.310668 14531 raft_consensus.cc:686] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 2 LEADER]: Becoming Leader. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Running, Role: LEADER
I20240503 08:14:56.311388 14531 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:56.316231 12090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a60d2ed0744f35be8054955dcf8aa0" candidate_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "925c339b047b4cf5bae840c5a357d3de"
I20240503 08:14:56.316570 12090 raft_consensus.cc:3037] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:56.316924 9387 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: term changed from 1 to 2, leader changed from 8de08e679a904ee48ba659861c730725 (127.24.123.3) to 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9). New cstate: current_term: 2 leader_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:56.319089 12090 raft_consensus.cc:2454] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a70e21b51a142bd8f7dc93fa58742fb in term 2.
I20240503 08:14:56.737206 9533 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:14:56.737875 14531 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:56.799557 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:14:56.800098 14531 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:56.807215 9798 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:14:56.807705 14531 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:56.832778 10056 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:14:56.833343 14531 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:56.839426 12090 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:14:56.841593 9667 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:14:56.843183 14531 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240503 08:14:56.844020 14531 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240503 08:14:56.877197 10319 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:14:56.877800 14559 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:14:57.407667 11957 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:14:57.411290 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:14:57.411471 10056 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:14:57.411576 9667 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:14:57.412063 14558 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:57.412146 12090 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:14:57.412439 14554 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:57.412734 14559 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:57.412997 9533 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:14:57.413218 10319 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:14:57.414562 14581 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:57.415534 14581 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:57.416889 14581 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:57.417944 14554 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:57.417969 9667 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:57.419539 10319 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:57.420147 9534 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:57.423745 12090 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:57.425848 9392 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 9 to 11, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New cstate: current_term: 2 leader_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:14:57.426015 10056 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:57.430095 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49a60d2ed0744f35be8054955dcf8aa0 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:57.431437 10189 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:14:57.445345 9777 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 with delete_type TABLET_DATA_TOMBSTONED (TS 8de08e679a904ee48ba659861c730725 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:51550
I20240503 08:14:57.446195 14587 tablet_replica.cc:330] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: stopping tablet replica
I20240503 08:14:57.446511 14587 raft_consensus.cc:2229] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:57.446943 14587 raft_consensus.cc:2258] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:57.448658 14587 ts_tablet_manager.cc:1875] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:57.453140 14587 ts_tablet_manager.cc:1888] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 08:14:57.453245 14587 log.cc:1196] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-2/wal/wals/49a60d2ed0744f35be8054955dcf8aa0
I20240503 08:14:57.453930 9378 catalog_manager.cc:4830] TS 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): tablet 49a60d2ed0744f35be8054955dcf8aa0 (table rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]) successfully deleted
I20240503 08:14:57.640935 9909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:57.663365 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:14:57.684412 11938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:57.701699 12070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.738224 9777 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:14:57.754801 10038 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:57.774367 12204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:57.792991 9513 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:57.827785 10169 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:57.841502 9646 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:58.096930 9909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:58.118953 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:14:58.121476 9777 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:14:58.121874 11938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:58.150354 12070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:58.164577 10038 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:58.190400 9513 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:14:58.225507 12204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:14:58.235107 9646 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:14:58.249176 10169 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:14:58.368732 11957 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } }
I20240503 08:14:58.368798 9534 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } }
I20240503 08:14:58.371838 12090 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:14:58.372385 14558 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:58.372529 10056 consensus_queue.cc:237] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
W20240503 08:14:58.372835 11891 consensus_peers.cc:485] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb -> Peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): Couldn't send request to peer 8de08e679a904ee48ba659861c730725. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.376435 10321 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:14:58.377081 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:14:58.377635 10057 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:14:58.377981 14554 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:58.378439 14709 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:58.379179 14554 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:58.379566 9533 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:14:58.381336 14558 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:58.381505 11958 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:14:58.382375 9993 consensus_peers.cc:485] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7247b6f7e08946709b215d2f6a26a2a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.382925 10321 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.383908 9667 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:14:58.384533 9928 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.385154 14639 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.385704 14108 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.386014 14639 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.387495 11958 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.387856 12222 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:14:58.388046 9471 consensus_peers.cc:485] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 -> Peer 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): Couldn't send request to peer 06f1b17731834350a289de93abab8542. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.388509 14453 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.389838 9667 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.390607 14453 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.391440 14108 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.394007 9795 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.394934 9533 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.395803 14554 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:58.397451 10057 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.397455 10189 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.397773 9661 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.399780 14639 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.400286 14639 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.404177 9388 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 11 to 12, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New cstate: current_term: 2 leader_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.409355 14558 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.429379 9532 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.433425 14711 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.438417 10319 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.440097 9661 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.441531 14453 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.444437 10188 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.445134 11957 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.446861 12090 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.448105 14523 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.448770 9795 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.450587 10320 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.451937 14523 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.455216 10321 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.455890 9387 catalog_manager.cc:5484] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.458477 9663 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.486047 10057 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.486753 14523 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:14:58.511817 9733 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.5:43643, user_credentials={real_user=slave}} blocked reactor thread for 58865us
I20240503 08:14:58.514000 9928 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.519675 14523 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:58.523124 10056 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:58.527674 11958 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:58.527999 10189 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:58.534427 9533 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.534399 10320 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:58.550073 12222 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:58.553089 9661 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:58.566103 9385 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 10 to 11, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:58.576072 9928 consensus_queue.cc:237] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:58.586828 10057 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.587553 11958 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.588078 10320 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.588768 14749 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.589128 14279 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.589748 14279 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.611052 9795 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.611706 14279 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.618690 9533 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.618803 9661 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.619426 14279 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.619246 14749 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.619993 14279 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.620035 10321 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
W20240503 08:14:58.623979 9862 consensus_peers.cc:485] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd41cbc6c69441c1af5c36c382cfe366. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.625909 9533 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } }
I20240503 08:14:58.629719 9798 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
W20240503 08:14:58.631532 9468 consensus_peers.cc:485] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): Couldn't send request to peer 8de08e679a904ee48ba659861c730725. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.632303 9393 catalog_manager.cc:5484] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "06f1b17731834350a289de93abab8542" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.633247 12090 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.633754 11957 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.637732 10056 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.639233 9661 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.642165 14453 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.643378 11958 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.647953 14740 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.656934 9928 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.657568 10321 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.657989 14740 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.658583 14523 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.661197 9661 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.661808 14453 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.663946 10056 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.665928 14453 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.673836 14453 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.675401 10320 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.679900 12090 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.680099 9661 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.680317 10056 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.686625 9389 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 10 to 11, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.685011 10057 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } }
I20240503 08:14:58.703430 9926 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.708078 11957 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } }
I20240503 08:14:58.713851 9534 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
W20240503 08:14:58.714514 9994 consensus_peers.cc:485] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 -> Peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Couldn't send request to peer fa60e870de844325bad4b9185eaee468. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.717772 12090 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.718189 12222 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.718398 9928 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.718502 14639 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.719040 14711 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.719336 14732 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.719576 10320 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.720723 9661 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.721199 14714 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.721498 14732 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.726439 9798 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.727108 14639 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.752584 14714 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:58.754295 12222 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:58.754559 10321 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:58.759860 9661 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:58.760828 9392 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf reported cstate change: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:58.783291 9798 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:58.784559 9928 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:58.786489 12090 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:58.837528 14790 ts_tablet_manager.cc:902] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:58.838470 10189 consensus_queue.cc:237] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:58.848940 10057 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } }
I20240503 08:14:58.853811 11958 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.854182 9797 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.854750 14480 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.855463 14480 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:58.857717 10124 consensus_peers.cc:485] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 -> Peer ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8:43883): Couldn't send request to peer ca5fa388d5b84a0abac1c05275155dcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9306edc6261425cb2c7bef9caf9135b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.860189 9534 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.860388 9661 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.860873 14579 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.861157 10321 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.861474 14480 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.863209 10056 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:58.865469 14480 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:14:58.872438 9993 consensus_peers.cc:485] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 -> Peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353): Couldn't send request to peer b3d3436263d04c57b2d47cc9d766efa5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:58.873454 12090 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.873773 11957 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.874477 14732 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.875048 14732 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.875772 14480 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:58.897858 9795 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.898360 9661 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.898576 14714 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.899109 14732 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.905357 14810 ts_tablet_manager.cc:902] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:58.909217 10320 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.910020 10189 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:58.910565 14714 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.911489 14714 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:58.924399 14714 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } } }
I20240503 08:14:58.926507 14793 ts_tablet_manager.cc:902] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Initiating tablet copy from peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619)
I20240503 08:14:58.922438 14480 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.937412 10189 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } } }
I20240503 08:14:58.939040 12089 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } } }
I20240503 08:14:58.940606 14810 tablet_copy_client.cc:308] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43643
I20240503 08:14:58.943015 10080 tablet_copy_service.cc:140] P a1402a75453d4ccf992606dcce77dc51: Received BeginTabletCopySession request for tablet 7247b6f7e08946709b215d2f6a26a2a4 from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:41803)
I20240503 08:14:58.943188 10080 tablet_copy_service.cc:161] P a1402a75453d4ccf992606dcce77dc51: Beginning new tablet copy session on tablet 7247b6f7e08946709b215d2f6a26a2a4 from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:41803: session id = ca5fa388d5b84a0abac1c05275155dcf-7247b6f7e08946709b215d2f6a26a2a4
I20240503 08:14:58.946039 10080 tablet_copy_source_session.cc:205] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:58.945665 9392 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de reported cstate change: config changed from index 10 to 11, NON_VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) added. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } } }
I20240503 08:14:58.947547 14810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7247b6f7e08946709b215d2f6a26a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.953085 14793 tablet_copy_client.cc:308] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.9:44619
I20240503 08:14:58.954156 11978 tablet_copy_service.cc:140] P 3a70e21b51a142bd8f7dc93fa58742fb: Received BeginTabletCopySession request for tablet 49a60d2ed0744f35be8054955dcf8aa0 from peer 8de08e679a904ee48ba659861c730725 ({username='slave'} at 127.24.123.3:33553)
I20240503 08:14:58.953749 9795 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } } }
I20240503 08:14:58.954408 11978 tablet_copy_service.cc:161] P 3a70e21b51a142bd8f7dc93fa58742fb: Beginning new tablet copy session on tablet 49a60d2ed0744f35be8054955dcf8aa0 from peer 8de08e679a904ee48ba659861c730725 at {username='slave'} at 127.24.123.3:33553: session id = 8de08e679a904ee48ba659861c730725-49a60d2ed0744f35be8054955dcf8aa0
I20240503 08:14:58.954473 11958 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } } }
I20240503 08:14:58.959239 10321 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.965668 11978 tablet_copy_source_session.cc:205] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:58.970013 14793 ts_tablet_manager.cc:1875] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:14:58.969323 9667 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.974668 14810 tablet_copy_client.cc:783] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:58.974969 14810 tablet_copy_client.cc:647] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:58.977584 9389 catalog_manager.cc:5484] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.981882 14810 tablet_copy_client.cc:523] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:58.982774 9534 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.983453 14793 ts_tablet_manager.cc:1888] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.10
I20240503 08:14:58.984627 14793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:58.987250 14810 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:58.991142 11958 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:58.992007 14793 tablet_copy_client.cc:783] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:58.992296 14793 tablet_copy_client.cc:647] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:58.994488 14793 tablet_copy_client.cc:523] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.006539 14793 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:59.018517 10056 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:59.028546 9798 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } } }
I20240503 08:14:59.044246 9661 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } } }
I20240503 08:14:59.065809 14790 tablet_copy_client.cc:308] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:59.066849 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 3fe97b80e1294c4791560daf991bfcdb from peer 06f1b17731834350a289de93abab8542 ({username='slave'} at 127.24.123.4:45237)
I20240503 08:14:59.067104 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 3fe97b80e1294c4791560daf991bfcdb from peer 06f1b17731834350a289de93abab8542 at {username='slave'} at 127.24.123.4:45237: session id = 06f1b17731834350a289de93abab8542-3fe97b80e1294c4791560daf991bfcdb
I20240503 08:14:59.070147 9555 tablet_copy_source_session.cc:205] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:59.084226 10321 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } } }
I20240503 08:14:59.105217 14790 ts_tablet_manager.cc:1875] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:14:59.112958 14830 ts_tablet_manager.cc:902] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer 06f1b17731834350a289de93abab8542 (127.24.123.4:38565)
I20240503 08:14:59.113190 10320 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } }
I20240503 08:14:59.134801 12090 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:59.135488 12222 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:59.136054 14725 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:14:59.137984 10256 consensus_peers.cc:485] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 -> Peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Couldn't send request to peer fa60e870de844325bad4b9185eaee468. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:59.139005 14725 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:59.147540 14844 ts_tablet_manager.cc:902] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Initiating tablet copy from peer a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:59.151337 14859 ts_tablet_manager.cc:902] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:14:59.154656 14790 ts_tablet_manager.cc:1888] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:14:59.155191 14790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe97b80e1294c4791560daf991bfcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.156967 9798 consensus_queue.cc:237] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:59.159572 9928 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:59.160671 14537 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:59.161113 9534 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:59.161768 9797 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:59.161718 14537 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:59.162325 14537 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:59.168536 10056 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:14:59.169471 14862 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:14:59.180042 14537 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
W20240503 08:14:59.185844 9731 consensus_peers.cc:485] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f13c7cfe35b3403f87940a124e6d9cda. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:59.187803 9533 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.187533 12222 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.187784 12090 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.188701 14638 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.189262 12223 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.189934 9389 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 reported cstate change: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:59.192235 14638 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.180425 9926 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.210063 9661 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.210652 14638 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.217542 14790 tablet_copy_client.cc:783] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.217883 14790 tablet_copy_client.cc:647] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.215638 10056 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.223086 10059 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.223384 10320 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.223973 14832 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.229173 9797 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.230105 14790 tablet_copy_client.cc:523] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.230912 14832 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.236217 14790 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:14:59.238477 9534 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.247160 9926 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.247814 14876 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.249420 14793 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:59.251354 14793 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Bootstrap complete.
I20240503 08:14:59.251060 14638 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.253118 9928 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.254544 14844 tablet_copy_client.cc:308] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43643
I20240503 08:14:59.255410 10080 tablet_copy_service.cc:140] P a1402a75453d4ccf992606dcce77dc51: Received BeginTabletCopySession request for tablet 0f6758eb87b047d29a060fcbfb775903 from peer fa60e870de844325bad4b9185eaee468 ({username='slave'} at 127.24.123.6:54911)
I20240503 08:14:59.255578 10080 tablet_copy_service.cc:161] P a1402a75453d4ccf992606dcce77dc51: Beginning new tablet copy session on tablet 0f6758eb87b047d29a060fcbfb775903 from peer fa60e870de844325bad4b9185eaee468 at {username='slave'} at 127.24.123.6:54911: session id = fa60e870de844325bad4b9185eaee468-0f6758eb87b047d29a060fcbfb775903
I20240503 08:14:59.256685 14810 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:59.258379 10080 tablet_copy_source_session.cc:205] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:59.261137 12223 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.261144 9661 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.258726 9389 catalog_manager.cc:5484] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.275017 14793 ts_tablet_manager.cc:1367] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.269s user 0.059s sys 0.006s
I20240503 08:14:59.277122 14844 ts_tablet_manager.cc:1875] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:14:59.277158 14793 raft_consensus.cc:348] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } }
I20240503 08:14:59.279163 14793 raft_consensus.cc:729] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: LEARNER
I20240503 08:14:59.281186 14793 consensus_queue.cc:260] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } }
I20240503 08:14:59.279425 9533 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.282061 14793 ts_tablet_manager.cc:1398] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:14:59.287297 11978 tablet_copy_service.cc:342] P 3a70e21b51a142bd8f7dc93fa58742fb: Request end of tablet copy session 8de08e679a904ee48ba659861c730725-49a60d2ed0744f35be8054955dcf8aa0 received from {username='slave'} at 127.24.123.3:33553
I20240503 08:14:59.287465 11978 tablet_copy_service.cc:433] P 3a70e21b51a142bd8f7dc93fa58742fb: ending tablet copy session 8de08e679a904ee48ba659861c730725-49a60d2ed0744f35be8054955dcf8aa0 on tablet 49a60d2ed0744f35be8054955dcf8aa0 with peer 8de08e679a904ee48ba659861c730725
I20240503 08:14:59.303611 9797 raft_consensus.cc:1203] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240503 08:14:59.311077 10059 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.312675 10321 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.317149 14810 tablet_bootstrap.cc:492] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:59.317487 14810 ts_tablet_manager.cc:1367] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.330s user 0.051s sys 0.024s
I20240503 08:14:59.317952 14830 tablet_copy_client.cc:308] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.4:38565
I20240503 08:14:59.323825 14859 tablet_copy_client.cc:308] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:14:59.324505 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 1be499fcc2bf4716b16a5983950d3ed8 from peer 8de08e679a904ee48ba659861c730725 ({username='slave'} at 127.24.123.3:53687)
I20240503 08:14:59.324666 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 1be499fcc2bf4716b16a5983950d3ed8 from peer 8de08e679a904ee48ba659861c730725 at {username='slave'} at 127.24.123.3:53687: session id = 8de08e679a904ee48ba659861c730725-1be499fcc2bf4716b16a5983950d3ed8
I20240503 08:14:59.318334 14810 raft_consensus.cc:348] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:59.325090 14810 raft_consensus.cc:729] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:59.327543 9555 tablet_copy_source_session.cc:205] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:59.325323 14810 consensus_queue.cc:260] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:59.330077 14810 ts_tablet_manager.cc:1398] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 08:14:59.331849 10080 tablet_copy_service.cc:342] P a1402a75453d4ccf992606dcce77dc51: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-7247b6f7e08946709b215d2f6a26a2a4 received from {username='slave'} at 127.24.123.8:41803
I20240503 08:14:59.331971 10080 tablet_copy_service.cc:433] P a1402a75453d4ccf992606dcce77dc51: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-7247b6f7e08946709b215d2f6a26a2a4 on tablet 7247b6f7e08946709b215d2f6a26a2a4 with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:59.333642 14859 ts_tablet_manager.cc:1875] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:14:59.334430 14896 ts_tablet_manager.cc:902] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: Initiating tablet copy from peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175)
I20240503 08:14:59.345185 9797 consensus_queue.cc:237] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } }
I20240503 08:14:59.348753 14844 ts_tablet_manager.cc:1888] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:14:59.349088 14844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6758eb87b047d29a060fcbfb775903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.350610 9949 tablet_copy_service.cc:140] P 06f1b17731834350a289de93abab8542: Received BeginTabletCopySession request for tablet dd41cbc6c69441c1af5c36c382cfe366 from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:50283)
I20240503 08:14:59.350786 9949 tablet_copy_service.cc:161] P 06f1b17731834350a289de93abab8542: Beginning new tablet copy session on tablet dd41cbc6c69441c1af5c36c382cfe366 from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:50283: session id = ca5fa388d5b84a0abac1c05275155dcf-dd41cbc6c69441c1af5c36c382cfe366
I20240503 08:14:59.353628 9949 tablet_copy_source_session.cc:205] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:59.354962 14844 tablet_copy_client.cc:783] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.355118 14830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd41cbc6c69441c1af5c36c382cfe366. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.355258 14844 tablet_copy_client.cc:647] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.357210 14844 tablet_copy_client.cc:523] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.359211 14859 ts_tablet_manager.cc:1888] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 08:14:59.361840 14859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.362396 14830 tablet_copy_client.cc:783] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.362658 14830 tablet_copy_client.cc:647] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.364444 14830 tablet_copy_client.cc:523] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.370995 10320 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:59.375154 14830 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:59.375876 12090 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:14:59.376454 9926 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:14:59.377379 10256 consensus_peers.cc:485] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 -> Peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Couldn't send request to peer fa60e870de844325bad4b9185eaee468. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:59.377619 10059 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:14:59.377563 14537 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.377789 14853 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.378742 14537 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.380666 14844 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:14:59.385941 14725 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.390221 12090 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.415416 14896 tablet_copy_client.cc:308] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.6:37175
I20240503 08:14:59.416117 10211 tablet_copy_service.cc:140] P fa60e870de844325bad4b9185eaee468: Received BeginTabletCopySession request for tablet c9306edc6261425cb2c7bef9caf9135b from peer ca5fa388d5b84a0abac1c05275155dcf ({username='slave'} at 127.24.123.8:40727)
I20240503 08:14:59.416311 10211 tablet_copy_service.cc:161] P fa60e870de844325bad4b9185eaee468: Beginning new tablet copy session on tablet c9306edc6261425cb2c7bef9caf9135b from peer ca5fa388d5b84a0abac1c05275155dcf at {username='slave'} at 127.24.123.8:40727: session id = ca5fa388d5b84a0abac1c05275155dcf-c9306edc6261425cb2c7bef9caf9135b
I20240503 08:14:59.419158 9926 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.419509 10059 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.427279 10211 tablet_copy_source_session.cc:205] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:59.428135 9389 catalog_manager.cc:5484] T e11a6197ccdd4e8d9a203efae4a94988 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 11 to 12, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New cstate: current_term: 1 leader_uuid: "0cb18b8b10f944e39829faf16e061fd6" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.431766 14896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9306edc6261425cb2c7bef9caf9135b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:14:59.443017 9734 consensus_peers.cc:485] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 -> Peer 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): Couldn't send request to peer 06f1b17731834350a289de93abab8542. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:59.445201 12223 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.445794 14876 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.449968 14790 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:59.452813 9661 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.452944 10189 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.453449 14790 tablet_bootstrap.cc:492] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Bootstrap complete.
I20240503 08:14:59.453473 14876 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.453781 14790 ts_tablet_manager.cc:1367] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.218s user 0.049s sys 0.011s
I20240503 08:14:59.454057 14832 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.454599 10321 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.456048 14832 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.459292 14896 tablet_copy_client.cc:783] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.459559 14896 tablet_copy_client.cc:647] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.461809 14896 tablet_copy_client.cc:523] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.456208 14790 raft_consensus.cc:348] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } }
I20240503 08:14:59.472930 14790 raft_consensus.cc:729] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: LEARNER
I20240503 08:14:59.473218 14790 consensus_queue.cc:260] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } }
I20240503 08:14:59.474833 14896 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap starting.
I20240503 08:14:59.475414 9533 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.476200 14790 ts_tablet_manager.cc:1398] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240503 08:14:59.476321 14832 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.478638 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session 06f1b17731834350a289de93abab8542-3fe97b80e1294c4791560daf991bfcdb received from {username='slave'} at 127.24.123.4:45237
I20240503 08:14:59.478772 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session 06f1b17731834350a289de93abab8542-3fe97b80e1294c4791560daf991bfcdb on tablet 3fe97b80e1294c4791560daf991bfcdb with peer 06f1b17731834350a289de93abab8542
I20240503 08:14:59.490029 10059 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:14:59.490712 14832 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:14:59.529356 14844 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:59.529831 14844 tablet_bootstrap.cc:492] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Bootstrap complete.
I20240503 08:14:59.530134 14844 ts_tablet_manager.cc:1367] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.151s user 0.042s sys 0.010s
I20240503 08:14:59.534381 12223 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:14:59.538146 9797 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:14:59.538743 9533 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:14:59.542083 14844 raft_consensus.cc:348] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } }
I20240503 08:14:59.542575 14844 raft_consensus.cc:729] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: LEARNER
I20240503 08:14:59.544306 14844 consensus_queue.cc:260] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } }
I20240503 08:14:59.544997 14832 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:59.547720 14844 ts_tablet_manager.cc:1398] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20240503 08:14:59.548403 10320 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:59.552551 10080 tablet_copy_service.cc:342] P a1402a75453d4ccf992606dcce77dc51: Request end of tablet copy session fa60e870de844325bad4b9185eaee468-0f6758eb87b047d29a060fcbfb775903 received from {username='slave'} at 127.24.123.6:54911
I20240503 08:14:59.552692 10080 tablet_copy_service.cc:433] P a1402a75453d4ccf992606dcce77dc51: ending tablet copy session fa60e870de844325bad4b9185eaee468-0f6758eb87b047d29a060fcbfb775903 on tablet 0f6758eb87b047d29a060fcbfb775903 with peer fa60e870de844325bad4b9185eaee468
I20240503 08:14:59.559365 12223 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:59.566577 9392 catalog_manager.cc:5484] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf reported cstate change: config changed from index 7 to 8, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:59.568971 9661 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:59.572384 10189 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:59.574067 10056 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:59.577474 14725 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.578157 14859 tablet_copy_client.cc:783] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.578465 14859 tablet_copy_client.cc:647] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.578415 9533 raft_consensus.cc:2932] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:14:59.580219 14859 tablet_copy_client.cc:523] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.585552 12223 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.599406 14725 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.600574 14725 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.601662 14903 ts_tablet_manager.cc:902] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Initiating tablet copy from peer a1402a75453d4ccf992606dcce77dc51 (127.24.123.5:43643)
I20240503 08:14:59.605757 9797 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
I20240503 08:14:59.608672 9534 raft_consensus.cc:2932] T e11a6197ccdd4e8d9a203efae4a94988 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } } }
W20240503 08:14:59.616403 10255 consensus_peers.cc:485] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 -> Peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): Couldn't send request to peer 3a70e21b51a142bd8f7dc93fa58742fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e11a6197ccdd4e8d9a203efae4a94988. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:59.624964 14859 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Bootstrap starting.
I20240503 08:14:59.626718 14903 tablet_copy_client.cc:308] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.5:43643
I20240503 08:14:59.633991 10080 tablet_copy_service.cc:140] P a1402a75453d4ccf992606dcce77dc51: Received BeginTabletCopySession request for tablet 43f3607501a84050962dc116210c09cd from peer b3d3436263d04c57b2d47cc9d766efa5 ({username='slave'} at 127.24.123.1:37591)
I20240503 08:14:59.634145 10080 tablet_copy_service.cc:161] P a1402a75453d4ccf992606dcce77dc51: Beginning new tablet copy session on tablet 43f3607501a84050962dc116210c09cd from peer b3d3436263d04c57b2d47cc9d766efa5 at {username='slave'} at 127.24.123.1:37591: session id = b3d3436263d04c57b2d47cc9d766efa5-43f3607501a84050962dc116210c09cd
I20240503 08:14:59.637189 10080 tablet_copy_source_session.cc:205] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:59.640923 14903 ts_tablet_manager.cc:1875] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:14:59.656255 14830 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:59.656723 14830 tablet_bootstrap.cc:492] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:59.657030 14830 ts_tablet_manager.cc:1367] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.283s user 0.096s sys 0.057s
I20240503 08:14:59.657809 14830 raft_consensus.cc:348] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:59.658138 14830 raft_consensus.cc:729] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:59.658365 14830 consensus_queue.cc:260] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:59.659056 14830 ts_tablet_manager.cc:1398] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:14:59.659806 9949 tablet_copy_service.cc:342] P 06f1b17731834350a289de93abab8542: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-dd41cbc6c69441c1af5c36c382cfe366 received from {username='slave'} at 127.24.123.8:50283
I20240503 08:14:59.659922 9949 tablet_copy_service.cc:433] P 06f1b17731834350a289de93abab8542: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-dd41cbc6c69441c1af5c36c382cfe366 on tablet dd41cbc6c69441c1af5c36c382cfe366 with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:59.674474 12223 raft_consensus.cc:1203] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:59.676345 9533 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } }
I20240503 08:14:59.680614 12090 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:14:59.681360 14712 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.683678 9926 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:14:59.684237 9469 consensus_peers.cc:485] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Couldn't send request to peer fa60e870de844325bad4b9185eaee468. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:59.686024 11957 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:14:59.687378 10056 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:14:59.687841 9468 consensus_peers.cc:485] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805): Couldn't send request to peer 8de08e679a904ee48ba659861c730725. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:59.688724 10320 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:14:59.689141 14712 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.689837 14712 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.690940 14453 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.695209 14453 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.697669 14939 ts_tablet_manager.cc:902] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:14:59.699450 14712 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.703725 12090 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.708830 11958 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.709705 10320 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.711546 14939 tablet_copy_client.cc:308] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:41805
I20240503 08:14:59.712432 9818 tablet_copy_service.cc:140] P 8de08e679a904ee48ba659861c730725: Received BeginTabletCopySession request for tablet f13c7cfe35b3403f87940a124e6d9cda from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:42981)
I20240503 08:14:59.712592 9818 tablet_copy_service.cc:161] P 8de08e679a904ee48ba659861c730725: Beginning new tablet copy session on tablet f13c7cfe35b3403f87940a124e6d9cda from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:42981: session id = 3a70e21b51a142bd8f7dc93fa58742fb-f13c7cfe35b3403f87940a124e6d9cda
I20240503 08:14:59.715502 9392 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 11 to 12, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.717841 10189 raft_consensus.cc:1203] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 08:14:59.722764 9818 tablet_copy_source_session.cc:205] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:59.724048 14939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c7cfe35b3403f87940a124e6d9cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.727124 9926 raft_consensus.cc:1203] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 08:14:59.729305 14939 tablet_copy_client.cc:783] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.729540 14939 tablet_copy_client.cc:647] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.732671 14888 raft_consensus.cc:1050] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: attempting to promote NON_VOTER 8de08e679a904ee48ba659861c730725 to VOTER
I20240503 08:14:59.733976 14888 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } }
I20240503 08:14:59.738760 14896 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:59.739323 14896 tablet_bootstrap.cc:492] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: Bootstrap complete.
I20240503 08:14:59.739702 14939 tablet_copy_client.cc:523] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.743464 14939 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:14:59.739603 14896 ts_tablet_manager.cc:1367] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: Time spent bootstrapping tablet: real 0.265s user 0.071s sys 0.049s
I20240503 08:14:59.748744 14896 raft_consensus.cc:348] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:59.748955 9797 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 LEARNER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:14:59.749083 14896 raft_consensus.cc:729] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5fa388d5b84a0abac1c05275155dcf, State: Initialized, Role: LEARNER
I20240503 08:14:59.749747 14709 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:14:59.753937 14896 consensus_queue.cc:260] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: NON_VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: true } }
I20240503 08:14:59.754684 14896 ts_tablet_manager.cc:1398] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:14:59.755872 10211 tablet_copy_service.cc:342] P fa60e870de844325bad4b9185eaee468: Request end of tablet copy session ca5fa388d5b84a0abac1c05275155dcf-c9306edc6261425cb2c7bef9caf9135b received from {username='slave'} at 127.24.123.8:40727
I20240503 08:14:59.756024 10211 tablet_copy_service.cc:433] P fa60e870de844325bad4b9185eaee468: ending tablet copy session ca5fa388d5b84a0abac1c05275155dcf-c9306edc6261425cb2c7bef9caf9135b on tablet c9306edc6261425cb2c7bef9caf9135b with peer ca5fa388d5b84a0abac1c05275155dcf
I20240503 08:14:59.759624 12090 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:14:59.760473 10320 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:14:59.760648 14709 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:14:59.761216 14709 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:14:59.762140 10056 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:14:59.780342 14961 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:14:59.784883 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:14:59.785442 14709 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:14:59.791473 14965 ts_tablet_manager.cc:902] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: Initiating tablet copy from peer 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:59.793298 14903 ts_tablet_manager.cc:1888] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 08:14:59.793803 14903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f3607501a84050962dc116210c09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.803364 9661 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:14:59.804406 9532 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:14:59.804756 14888 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:14:59.806280 14963 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:14:59.808655 9926 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.812419 9667 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:14:59.816066 14453 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:14:59.819489 12223 raft_consensus.cc:1203] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:14:59.829157 14961 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.829316 10057 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.833700 14903 tablet_copy_client.cc:783] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.833941 10058 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.839663 14903 tablet_copy_client.cc:647] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.840760 9661 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.851243 10189 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.854771 14903 tablet_copy_client.cc:523] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.856679 9392 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 reported cstate change: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.857458 12090 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.860375 14903 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap starting.
I20240503 08:14:59.867107 10320 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.886041 9797 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.890257 14939 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:59.890748 14939 tablet_bootstrap.cc:492] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:14:59.897778 14939 ts_tablet_manager.cc:1367] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.154s user 0.061s sys 0.036s
I20240503 08:14:59.898594 14939 raft_consensus.cc:348] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:59.898973 14939 raft_consensus.cc:729] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:14:59.899194 14939 consensus_queue.cc:260] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:14:59.901997 14939 ts_tablet_manager.cc:1398] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:14:59.902630 9818 tablet_copy_service.cc:342] P 8de08e679a904ee48ba659861c730725: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-f13c7cfe35b3403f87940a124e6d9cda received from {username='slave'} at 127.24.123.9:42981
I20240503 08:14:59.902745 9818 tablet_copy_service.cc:433] P 8de08e679a904ee48ba659861c730725: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-f13c7cfe35b3403f87940a124e6d9cda on tablet f13c7cfe35b3403f87940a124e6d9cda with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:14:59.902388 9667 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:14:59.905333 9532 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.913314 11958 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } }
I20240503 08:14:59.918435 12090 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 08:14:59.919266 14961 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:14:59.923645 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 08:14:59.923697 10057 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 08:14:59.924407 14963 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:14:59.924679 10321 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 08:14:59.924997 14963 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:14:59.925679 9532 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 08:14:59.925705 14961 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:14:59.928447 9797 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 08:14:59.928802 14961 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:14:59.929131 14888 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.933492 10189 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.934789 14963 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:14:59.934260 12089 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.936725 10321 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.938503 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49a60d2ed0744f35be8054955dcf8aa0 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:14:59.940894 9532 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.941315 10057 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.947818 9394 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 13 to 14, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New cstate: current_term: 2 leader_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:59.949199 9797 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } }
I20240503 08:14:59.955261 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a60d2ed0744f35be8054955dcf8aa0 with delete_type TABLET_DATA_TOMBSTONED (TS d1eb5e69c6264a20a6e8297cc5deb8d1 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:55050
I20240503 08:14:59.963359 14965 tablet_copy_client.cc:308] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.7:42473
I20240503 08:14:59.964052 10342 tablet_copy_service.cc:140] P 0cb18b8b10f944e39829faf16e061fd6: Received BeginTabletCopySession request for tablet e11a6197ccdd4e8d9a203efae4a94988 from peer 3a70e21b51a142bd8f7dc93fa58742fb ({username='slave'} at 127.24.123.9:58419)
I20240503 08:14:59.964221 10342 tablet_copy_service.cc:161] P 0cb18b8b10f944e39829faf16e061fd6: Beginning new tablet copy session on tablet e11a6197ccdd4e8d9a203efae4a94988 from peer 3a70e21b51a142bd8f7dc93fa58742fb at {username='slave'} at 127.24.123.9:58419: session id = 3a70e21b51a142bd8f7dc93fa58742fb-e11a6197ccdd4e8d9a203efae4a94988
I20240503 08:14:59.969342 10342 tablet_copy_source_session.cc:205] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:59.971079 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:59.978332 14965 tablet_copy_client.cc:783] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:59.978708 14965 tablet_copy_client.cc:647] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:59.983286 14983 tablet_replica.cc:330] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1: stopping tablet replica
I20240503 08:14:59.983667 14983 raft_consensus.cc:2229] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:59.984848 14844 ts_tablet_manager.cc:902] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Initiating tablet copy from peer 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473)
I20240503 08:14:59.985252 14983 raft_consensus.cc:2258] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:59.985416 14965 tablet_copy_client.cc:523] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:59.988086 14983 ts_tablet_manager.cc:1875] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:14:59.989681 14965 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap starting.
I20240503 08:15:00.004029 14983 ts_tablet_manager.cc:1888] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240503 08:15:00.006034 14983 log.cc:1196] T 49a60d2ed0744f35be8054955dcf8aa0 P d1eb5e69c6264a20a6e8297cc5deb8d1: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-1/wal/wals/49a60d2ed0744f35be8054955dcf8aa0
I20240503 08:15:00.007133 9380 catalog_manager.cc:4830] TS d1eb5e69c6264a20a6e8297cc5deb8d1 (127.24.123.2:36123): tablet 49a60d2ed0744f35be8054955dcf8aa0 (table rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]) successfully deleted
I20240503 08:15:00.016594 14844 tablet_copy_client.cc:308] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.7:42473
I20240503 08:15:00.017242 10342 tablet_copy_service.cc:140] P 0cb18b8b10f944e39829faf16e061fd6: Received BeginTabletCopySession request for tablet e11a6197ccdd4e8d9a203efae4a94988 from peer fa60e870de844325bad4b9185eaee468 ({username='slave'} at 127.24.123.6:44357)
I20240503 08:15:00.017454 10342 tablet_copy_service.cc:161] P 0cb18b8b10f944e39829faf16e061fd6: Beginning new tablet copy session on tablet e11a6197ccdd4e8d9a203efae4a94988 from peer fa60e870de844325bad4b9185eaee468 at {username='slave'} at 127.24.123.6:44357: session id = fa60e870de844325bad4b9185eaee468-e11a6197ccdd4e8d9a203efae4a94988
I20240503 08:15:00.022006 10342 tablet_copy_source_session.cc:205] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:00.030092 14844 ts_tablet_manager.cc:1875] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:00.044945 11958 consensus_queue.cc:237] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } }
W20240503 08:15:00.052021 11894 consensus_peers.cc:485] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb -> Peer 06f1b17731834350a289de93abab8542 (127.24.123.4:38565): Couldn't send request to peer 06f1b17731834350a289de93abab8542. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:00.052237 9797 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 08:15:00.052676 12089 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 08:15:00.053786 10321 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 08:15:00.054505 14709 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:00.055018 14709 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:00.056116 14961 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:00.061442 10189 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 08:15:00.062237 14888 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:00.063556 10057 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 08:15:00.064507 14859 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:00.064924 14859 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Bootstrap complete.
I20240503 08:15:00.065237 14859 ts_tablet_manager.cc:1367] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Time spent bootstrapping tablet: real 0.440s user 0.048s sys 0.007s
I20240503 08:15:00.065431 14888 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:00.066026 14859 raft_consensus.cc:348] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } }
I20240503 08:15:00.066336 14859 raft_consensus.cc:729] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de08e679a904ee48ba659861c730725, State: Initialized, Role: LEARNER
I20240503 08:15:00.066538 14859 consensus_queue.cc:260] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } }
I20240503 08:15:00.067189 14859 ts_tablet_manager.cc:1398] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:15:00.066583 10189 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P fa60e870de844325bad4b9185eaee468 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:15:00.067469 9532 raft_consensus.cc:1261] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Refusing update from remote peer 3a70e21b51a142bd8f7dc93fa58742fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 08:15:00.069821 10057 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P a1402a75453d4ccf992606dcce77dc51 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:15:00.073525 9555 tablet_copy_service.cc:342] P b3d3436263d04c57b2d47cc9d766efa5: Request end of tablet copy session 8de08e679a904ee48ba659861c730725-1be499fcc2bf4716b16a5983950d3ed8 received from {username='slave'} at 127.24.123.3:53687
I20240503 08:15:00.073665 9555 tablet_copy_service.cc:433] P b3d3436263d04c57b2d47cc9d766efa5: ending tablet copy session 8de08e679a904ee48ba659861c730725-1be499fcc2bf4716b16a5983950d3ed8 on tablet 1be499fcc2bf4716b16a5983950d3ed8 with peer 8de08e679a904ee48ba659861c730725
I20240503 08:15:00.078233 9797 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:15:00.082942 14963 consensus_queue.cc:1035] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:15:00.083686 14998 ts_tablet_manager.cc:902] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Initiating tablet copy from peer 8de08e679a904ee48ba659861c730725 (127.24.123.3:41805)
I20240503 08:15:00.065426 14709 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:15:00.086763 9392 catalog_manager.cc:5484] T 49a60d2ed0744f35be8054955dcf8aa0 P 8de08e679a904ee48ba659861c730725 reported cstate change: config changed from index 14 to 15, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New cstate: current_term: 2 leader_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:15:00.089171 9532 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P b3d3436263d04c57b2d47cc9d766efa5 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:15:00.094014 12089 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 925c339b047b4cf5bae840c5a357d3de [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:15:00.111963 10321 raft_consensus.cc:2932] T 49a60d2ed0744f35be8054955dcf8aa0 P 0cb18b8b10f944e39829faf16e061fd6 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 06f1b17731834350a289de93abab8542 (127.24.123.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } } }
I20240503 08:15:00.125315 14903 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:00.125838 14903 tablet_bootstrap.cc:492] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Bootstrap complete.
I20240503 08:15:00.126164 14903 ts_tablet_manager.cc:1367] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Time spent bootstrapping tablet: real 0.266s user 0.053s sys 0.000s
I20240503 08:15:00.134373 14903 raft_consensus.cc:348] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } }
I20240503 08:15:00.134747 14903 raft_consensus.cc:729] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3d3436263d04c57b2d47cc9d766efa5, State: Initialized, Role: LEARNER
I20240503 08:15:00.135007 14903 consensus_queue.cc:260] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: NON_VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: true } }
I20240503 08:15:00.137393 14903 ts_tablet_manager.cc:1398] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240503 08:15:00.138317 10080 tablet_copy_service.cc:342] P a1402a75453d4ccf992606dcce77dc51: Request end of tablet copy session b3d3436263d04c57b2d47cc9d766efa5-43f3607501a84050962dc116210c09cd received from {username='slave'} at 127.24.123.1:37591
I20240503 08:15:00.138445 10080 tablet_copy_service.cc:433] P a1402a75453d4ccf992606dcce77dc51: ending tablet copy session b3d3436263d04c57b2d47cc9d766efa5-43f3607501a84050962dc116210c09cd on tablet 43f3607501a84050962dc116210c09cd with peer b3d3436263d04c57b2d47cc9d766efa5
I20240503 08:15:00.145675 14844 ts_tablet_manager.cc:1888] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:15:00.145931 14844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a6197ccdd4e8d9a203efae4a94988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.150749 12223 raft_consensus.cc:1203] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:15:00.154367 14844 tablet_copy_client.cc:783] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:00.154605 14844 tablet_copy_client.cc:647] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:00.156996 14844 tablet_copy_client.cc:523] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:00.160698 14844 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:15:00.172662 14998 tablet_copy_client.cc:308] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.3:41805
I20240503 08:15:00.173537 9818 tablet_copy_service.cc:140] P 8de08e679a904ee48ba659861c730725: Received BeginTabletCopySession request for tablet 34be73cc40264224ad798d7518c45ed1 from peer 06f1b17731834350a289de93abab8542 ({username='slave'} at 127.24.123.4:44589)
I20240503 08:15:00.173717 9818 tablet_copy_service.cc:161] P 8de08e679a904ee48ba659861c730725: Beginning new tablet copy session on tablet 34be73cc40264224ad798d7518c45ed1 from peer 06f1b17731834350a289de93abab8542 at {username='slave'} at 127.24.123.4:44589: session id = 06f1b17731834350a289de93abab8542-34be73cc40264224ad798d7518c45ed1
I20240503 08:15:00.176822 9818 tablet_copy_source_session.cc:205] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:00.177963 14998 ts_tablet_manager.cc:1875] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:00.186182 14998 ts_tablet_manager.cc:1888] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:15:00.187008 9797 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: NON_VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.189651 14711 raft_consensus.cc:1050] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:15:00.190783 14998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be73cc40264224ad798d7518c45ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.190797 14711 consensus_queue.cc:237] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:15:00.194736 14998 tablet_copy_client.cc:783] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:00.195097 14998 tablet_copy_client.cc:647] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:00.197175 14732 raft_consensus.cc:1050] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51: attempting to promote NON_VOTER fa60e870de844325bad4b9185eaee468 to VOTER
I20240503 08:15:00.197397 14998 tablet_copy_client.cc:523] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:00.198376 14732 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } }
I20240503 08:15:00.201292 14998 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:15:00.206821 11958 raft_consensus.cc:1203] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:15:00.208833 9667 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.209336 12223 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.209903 9926 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.209898 9532 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.210441 14711 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.210916 14639 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.211150 10321 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.211781 11958 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.212917 14639 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.213049 9797 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.213438 14639 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.234483 14639 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.235334 14639 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.235335 9532 consensus_queue.cc:237] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } }
I20240503 08:15:00.237625 14712 raft_consensus.cc:1050] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER 06f1b17731834350a289de93abab8542 to VOTER
I20240503 08:15:00.238770 14712 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } }
I20240503 08:15:00.254328 14965 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:00.254817 14965 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: Bootstrap complete.
I20240503 08:15:00.255146 14965 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent bootstrapping tablet: real 0.266s user 0.070s sys 0.000s
I20240503 08:15:00.255942 14965 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:15:00.256282 14965 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a70e21b51a142bd8f7dc93fa58742fb, State: Initialized, Role: LEARNER
I20240503 08:15:00.256506 14965 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:15:00.257906 14965 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:15:00.258234 12089 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.258268 10189 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 LEARNER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.259361 10342 tablet_copy_service.cc:342] P 0cb18b8b10f944e39829faf16e061fd6: Request end of tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-e11a6197ccdd4e8d9a203efae4a94988 received from {username='slave'} at 127.24.123.9:58419
I20240503 08:15:00.259496 10342 tablet_copy_service.cc:433] P 0cb18b8b10f944e39829faf16e061fd6: ending tablet copy session 3a70e21b51a142bd8f7dc93fa58742fb-e11a6197ccdd4e8d9a203efae4a94988 on tablet e11a6197ccdd4e8d9a203efae4a94988 with peer 3a70e21b51a142bd8f7dc93fa58742fb
I20240503 08:15:00.259553 14639 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.260118 14639 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.270006 12090 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:15:00.270011 9469 consensus_peers.cc:485] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Couldn't send request to peer fa60e870de844325bad4b9185eaee468. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:00.270821 9926 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:15:00.271427 12222 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:15:00.271737 10320 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:15:00.272368 14453 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:00.273568 14941 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:00.273869 14453 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:00.274430 14947 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:00.276087 9926 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.276335 9797 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:15:00.276809 14941 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.277940 10320 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.278584 14711 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.279522 14947 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:00.280230 12222 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.280716 10057 raft_consensus.cc:1261] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:15:00.288452 14711 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.289592 14947 consensus_queue.cc:1035] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:15:00.290048 14711 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.290829 14711 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.297233 9928 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.302981 9392 catalog_manager.cc:5484] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.303620 14799 raft_consensus.cc:1050] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:15:00.304733 14799 consensus_queue.cc:237] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:15:00.318701 9667 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.321750 12223 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.322091 9661 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.322305 10191 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.322487 14947 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.323125 14941 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.324085 10321 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.324905 14947 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.324985 11958 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.325716 14453 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.326306 14453 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.330941 9928 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.331645 14639 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.334841 9663 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.335459 14639 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.346082 12222 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.346688 11956 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.349846 9666 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.350630 9795 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.350699 14937 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.351326 14937 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.351727 14822 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.363266 14947 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 06f1b17731834350a289de93abab8542 (127.24.123.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:00.364375 11957 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 06f1b17731834350a289de93abab8542 (127.24.123.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:00.369933 10191 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 06f1b17731834350a289de93abab8542 (127.24.123.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:00.370262 9386 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 11 to 12, 06f1b17731834350a289de93abab8542 (127.24.123.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:00.370797 10320 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 06f1b17731834350a289de93abab8542 (127.24.123.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:00.372277 9927 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 06f1b17731834350a289de93abab8542 (127.24.123.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:00.373945 12223 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 06f1b17731834350a289de93abab8542 (127.24.123.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:00.379772 10057 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.380628 14941 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.381330 14822 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.382517 9661 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 06f1b17731834350a289de93abab8542 (127.24.123.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:00.386368 12224 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.396927 10320 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.398272 9797 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.400810 11957 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.401357 9795 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.402514 9385 catalog_manager.cc:5484] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:15:00.406769 9534 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.410789 14711 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.416719 10058 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:00.418401 14844 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:00.418848 14844 tablet_bootstrap.cc:492] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Bootstrap complete.
I20240503 08:15:00.419194 9534 raft_consensus.cc:1203] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 08:15:00.419371 14844 ts_tablet_manager.cc:1367] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Time spent bootstrapping tablet: real 0.259s user 0.055s sys 0.007s
I20240503 08:15:00.423081 14844 raft_consensus.cc:348] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:15:00.422109 9798 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.423424 14844 raft_consensus.cc:729] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa60e870de844325bad4b9185eaee468, State: Initialized, Role: LEARNER
I20240503 08:15:00.425247 14844 consensus_queue.cc:260] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:15:00.426044 15047 ts_tablet_manager.cc:902] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:15:00.425956 14844 ts_tablet_manager.cc:1398] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20240503 08:15:00.429611 10124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.9:44619, user_credentials={real_user=slave}} blocked reactor thread for 79089.5us
I20240503 08:15:00.430606 10342 tablet_copy_service.cc:342] P 0cb18b8b10f944e39829faf16e061fd6: Request end of tablet copy session fa60e870de844325bad4b9185eaee468-e11a6197ccdd4e8d9a203efae4a94988 received from {username='slave'} at 127.24.123.6:44357
I20240503 08:15:00.430721 10320 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.430752 10342 tablet_copy_service.cc:433] P 0cb18b8b10f944e39829faf16e061fd6: ending tablet copy session fa60e870de844325bad4b9185eaee468-e11a6197ccdd4e8d9a203efae4a94988 on tablet e11a6197ccdd4e8d9a203efae4a94988 with peer fa60e870de844325bad4b9185eaee468
I20240503 08:15:00.431145 14937 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.432871 9534 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.442399 14941 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:00.442541 10320 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.443707 12223 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.429464 9926 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.473338 14639 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fa60e870de844325bad4b9185eaee468 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.480680 9666 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fa60e870de844325bad4b9185eaee468 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.482565 14998 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:15:00.483161 14998 tablet_bootstrap.cc:492] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Bootstrap complete.
I20240503 08:15:00.483105 10188 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fa60e870de844325bad4b9185eaee468 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.484694 14998 ts_tablet_manager.cc:1367] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Time spent bootstrapping tablet: real 0.282s user 0.061s sys 0.025s
I20240503 08:15:00.484552 9928 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fa60e870de844325bad4b9185eaee468 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.494302 14937 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.495330 14799 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.494944 14937 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.496902 11957 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.497532 12089 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fa60e870de844325bad4b9185eaee468 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.500378 9386 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 reported cstate change: config changed from index 11 to 12, fa60e870de844325bad4b9185eaee468 (127.24.123.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.501170 9661 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.502080 12090 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.513453 9386 catalog_manager.cc:5484] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.519358 14998 raft_consensus.cc:348] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } }
I20240503 08:15:00.524844 10319 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fa60e870de844325bad4b9185eaee468 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.522003 14998 raft_consensus.cc:729] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06f1b17731834350a289de93abab8542, State: Initialized, Role: LEARNER
I20240503 08:15:00.530185 14998 consensus_queue.cc:260] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: NON_VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: true } }
I20240503 08:15:00.531729 10059 raft_consensus.cc:2932] T 9428100e4a364805863419b22ca3c682 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER fa60e870de844325bad4b9185eaee468 (127.24.123.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.535179 10057 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.537528 12224 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.540377 9795 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.544860 10056 consensus_queue.cc:237] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:15:00.548712 10058 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 06f1b17731834350a289de93abab8542 (127.24.123.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:00.546016 14998 ts_tablet_manager.cc:1398] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542: Time spent starting tablet: real 0.061s user 0.002s sys 0.001s
I20240503 08:15:00.549500 10321 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.550068 9818 tablet_copy_service.cc:342] P 8de08e679a904ee48ba659861c730725: Request end of tablet copy session 06f1b17731834350a289de93abab8542-34be73cc40264224ad798d7518c45ed1 received from {username='slave'} at 127.24.123.4:44589
I20240503 08:15:00.550185 9818 tablet_copy_service.cc:433] P 8de08e679a904ee48ba659861c730725: ending tablet copy session 06f1b17731834350a289de93abab8542-34be73cc40264224ad798d7518c45ed1 on tablet 34be73cc40264224ad798d7518c45ed1 with peer 06f1b17731834350a289de93abab8542
I20240503 08:15:00.563678 9797 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fa60e870de844325bad4b9185eaee468 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.567502 9532 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.580559 12224 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.581300 9797 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.581331 9533 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.581585 15028 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.581662 12224 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fa60e870de844325bad4b9185eaee468 (127.24.123.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.582970 15028 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.583642 15028 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.584673 10321 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.585842 9928 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.589318 15028 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.589910 15050 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.595082 10189 raft_consensus.cc:1203] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 08:15:00.597644 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7247b6f7e08946709b215d2f6a26a2a4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:15:00.599486 14749 raft_consensus.cc:1050] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542: attempting to promote NON_VOTER ca5fa388d5b84a0abac1c05275155dcf to VOTER
I20240503 08:15:00.600507 14749 consensus_queue.cc:237] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:15:00.603206 9666 raft_consensus.cc:1261] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.605479 14732 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.605799 9927 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.607162 9666 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.610631 14712 raft_consensus.cc:1050] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: attempting to promote NON_VOTER 8de08e679a904ee48ba659861c730725 to VOTER
I20240503 08:15:00.610965 14732 consensus_queue.cc:1035] T 7247b6f7e08946709b215d2f6a26a2a4 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.611652 14712 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } }
I20240503 08:15:00.612452 9393 catalog_manager.cc:5484] T 7247b6f7e08946709b215d2f6a26a2a4 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.612555 12224 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.616943 11938 tablet_service.cc:1514] Processing DeleteTablet for tablet 7247b6f7e08946709b215d2f6a26a2a4 with delete_type TABLET_DATA_TOMBSTONED (TS 3a70e21b51a142bd8f7dc93fa58742fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57252
W20240503 08:15:00.617830 9469 consensus_peers.cc:485] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Couldn't send request to peer fa60e870de844325bad4b9185eaee468. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:00.618176 9534 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.620290 9795 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 LEARNER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.620922 14947 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.622462 10321 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.623247 14947 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.625923 9661 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.626431 14947 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.628044 9797 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.629132 12090 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.629617 14947 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.632671 9928 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.633529 14947 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.633215 10319 raft_consensus.cc:2932] T 7247b6f7e08946709b215d2f6a26a2a4 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.638486 14941 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.639008 10056 consensus_queue.cc:237] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } }
I20240503 08:15:00.640727 10058 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.642320 15092 tablet_replica.cc:330] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: stopping tablet replica
I20240503 08:15:00.642922 15092 raft_consensus.cc:2229] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:00.643255 15092 raft_consensus.cc:2258] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:00.643685 12222 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf [term 1 LEARNER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.644462 15002 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.647258 9927 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.648154 14732 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.648232 15047 tablet_copy_client.cc:308] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:15:00.647838 9385 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:15:00.648489 10188 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.649138 15028 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.648990 9663 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.649935 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 1be499fcc2bf4716b16a5983950d3ed8 from peer fa60e870de844325bad4b9185eaee468 ({username='slave'} at 127.24.123.6:51619)
I20240503 08:15:00.650079 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 1be499fcc2bf4716b16a5983950d3ed8 from peer fa60e870de844325bad4b9185eaee468 at {username='slave'} at 127.24.123.6:51619: session id = fa60e870de844325bad4b9185eaee468-1be499fcc2bf4716b16a5983950d3ed8
I20240503 08:15:00.649541 12088 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.652917 9555 tablet_copy_source_session.cc:205] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:00.654148 15047 ts_tablet_manager.cc:1875] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:00.656419 11957 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.657029 15002 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.660154 9927 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.662611 14941 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.668532 10319 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.668727 9661 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.669243 15002 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.669365 14639 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.672312 15028 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.674085 12224 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.674225 9797 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.666744 9795 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.676652 9380 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f6758eb87b047d29a060fcbfb775903 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:15:00.678362 9663 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.679920 10056 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.680204 9667 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.680398 15050 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.680876 15002 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.681272 15050 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.682329 9797 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.683594 15002 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.683125 10056 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.684082 12090 raft_consensus.cc:1261] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.684638 15050 consensus_queue.cc:1035] T 0f6758eb87b047d29a060fcbfb775903 P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.685777 10321 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.686189 12089 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.688795 15002 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.689707 9798 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.690304 12224 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.694257 9385 catalog_manager.cc:5484] T 0f6758eb87b047d29a060fcbfb775903 P d1eb5e69c6264a20a6e8297cc5deb8d1 reported cstate change: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.700263 10301 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6758eb87b047d29a060fcbfb775903 with delete_type TABLET_DATA_TOMBSTONED (TS 0cb18b8b10f944e39829faf16e061fd6 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:57798
I20240503 08:15:00.701294 14998 ts_tablet_manager.cc:902] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: Initiating tablet copy from peer 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619)
I20240503 08:15:00.702209 11958 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:00.703173 14941 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:00.705201 11958 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8de08e679a904ee48ba659861c730725 (127.24.123.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:00.707007 15092 ts_tablet_manager.cc:1875] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:00.712284 9534 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.713066 14749 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.677760 9928 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.723682 15002 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.728786 15100 tablet_replica.cc:330] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6: stopping tablet replica
I20240503 08:15:00.729243 9798 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.732080 9393 catalog_manager.cc:5484] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06f1b17731834350a289de93abab8542" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:15:00.733685 10320 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.734616 12222 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.740613 11958 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.740104 9534 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.744308 15100 raft_consensus.cc:2229] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:00.748150 9666 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.764143 15100 raft_consensus.cc:2258] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:00.766114 10056 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.772846 9927 consensus_queue.cc:237] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:15:00.777216 12224 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.778152 14999 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.778405 14844 ts_tablet_manager.cc:902] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: Initiating tablet copy from peer b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1:39353)
I20240503 08:15:00.779807 10321 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.780565 10188 raft_consensus.cc:2932] T 0f6758eb87b047d29a060fcbfb775903 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } }
I20240503 08:15:00.781736 14999 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.783068 9798 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.783661 14999 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.784478 9534 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.784487 9666 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.785117 15002 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.785354 14999 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.786082 14999 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.788722 9534 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.792568 15047 ts_tablet_manager.cc:1888] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:15:00.793896 15047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be499fcc2bf4716b16a5983950d3ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.794767 9386 catalog_manager.cc:5484] T dd41cbc6c69441c1af5c36c382cfe366 P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New cstate: current_term: 1 leader_uuid: "06f1b17731834350a289de93abab8542" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.795293 9667 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.802134 14638 raft_consensus.cc:1050] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725: attempting to promote NON_VOTER 3a70e21b51a142bd8f7dc93fa58742fb to VOTER
I20240503 08:15:00.802305 15092 ts_tablet_manager.cc:1888] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:15:00.803325 14638 consensus_queue.cc:237] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:15:00.809823 15047 tablet_copy_client.cc:783] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:00.814651 10056 raft_consensus.cc:1261] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 06f1b17731834350a289de93abab8542: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:00.804462 11938 tablet_service.cc:1514] Processing DeleteTablet for tablet dd41cbc6c69441c1af5c36c382cfe366 with delete_type TABLET_DATA_TOMBSTONED (TS 3a70e21b51a142bd8f7dc93fa58742fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57252
I20240503 08:15:00.817157 15092 log.cc:1196] T 7247b6f7e08946709b215d2f6a26a2a4 P 3a70e21b51a142bd8f7dc93fa58742fb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal/wals/7247b6f7e08946709b215d2f6a26a2a4
I20240503 08:15:00.817438 14998 tablet_copy_client.cc:308] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.9:44619
I20240503 08:15:00.817471 9798 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.818703 15002 consensus_queue.cc:1035] T dd41cbc6c69441c1af5c36c382cfe366 P 06f1b17731834350a289de93abab8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:00.819679 11978 tablet_copy_service.cc:140] P 3a70e21b51a142bd8f7dc93fa58742fb: Received BeginTabletCopySession request for tablet 49a60d2ed0744f35be8054955dcf8aa0 from peer 06f1b17731834350a289de93abab8542 ({username='slave'} at 127.24.123.4:46897)
I20240503 08:15:00.819885 11978 tablet_copy_service.cc:161] P 3a70e21b51a142bd8f7dc93fa58742fb: Beginning new tablet copy session on tablet 49a60d2ed0744f35be8054955dcf8aa0 from peer 06f1b17731834350a289de93abab8542 at {username='slave'} at 127.24.123.4:46897: session id = 06f1b17731834350a289de93abab8542-49a60d2ed0744f35be8054955dcf8aa0
I20240503 08:15:00.820904 10056 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.822700 11978 tablet_copy_source_session.cc:205] T 49a60d2ed0744f35be8054955dcf8aa0 P 3a70e21b51a142bd8f7dc93fa58742fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:00.826488 15047 tablet_copy_client.cc:647] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:00.829025 11957 raft_consensus.cc:1203] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 08:15:00.829421 14998 ts_tablet_manager.cc:1875] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:00.831482 15092 tablet_replica.cc:330] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: stopping tablet replica
I20240503 08:15:00.831768 15092 raft_consensus.cc:2229] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:00.832691 15092 raft_consensus.cc:2258] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:00.835700 9378 catalog_manager.cc:4830] TS 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): tablet 7247b6f7e08946709b215d2f6a26a2a4 (table rebalance_test_table_6 [id=c72b597b83114c4e8c9b97c776766268]) successfully deleted
I20240503 08:15:00.836687 15092 ts_tablet_manager.cc:1875] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:00.837973 9381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd41cbc6c69441c1af5c36c382cfe366 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:15:00.838927 12224 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.839236 10321 raft_consensus.cc:2932] T dd41cbc6c69441c1af5c36c382cfe366 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:00.853443 15100 ts_tablet_manager.cc:1875] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:00.857084 15047 tablet_copy_client.cc:523] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:00.866845 14844 tablet_copy_client.cc:308] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: tablet copy: Beginning tablet copy session from remote peer at address 127.24.123.1:39353
I20240503 08:15:00.869966 9555 tablet_copy_service.cc:140] P b3d3436263d04c57b2d47cc9d766efa5: Received BeginTabletCopySession request for tablet 9428100e4a364805863419b22ca3c682 from peer fa60e870de844325bad4b9185eaee468 ({username='slave'} at 127.24.123.6:51619)
I20240503 08:15:00.870122 9555 tablet_copy_service.cc:161] P b3d3436263d04c57b2d47cc9d766efa5: Beginning new tablet copy session on tablet 9428100e4a364805863419b22ca3c682 from peer fa60e870de844325bad4b9185eaee468 at {username='slave'} at 127.24.123.6:51619: session id = fa60e870de844325bad4b9185eaee468-9428100e4a364805863419b22ca3c682
I20240503 08:15:00.873191 9555 tablet_copy_source_session.cc:205] T 9428100e4a364805863419b22ca3c682 P b3d3436263d04c57b2d47cc9d766efa5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:15:00.875640 14844 ts_tablet_manager.cc:1875] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:15:00.875675 11957 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.876760 14832 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.886848 12224 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.887235 15047 tablet_bootstrap.cc:492] T 1be499fcc2bf4716b16a5983950d3ed8 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:15:00.887449 9534 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.888458 14832 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.888991 14832 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.895195 14998 ts_tablet_manager.cc:1888] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:15:00.895496 14998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a60d2ed0744f35be8054955dcf8aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.893850 14844 ts_tablet_manager.cc:1888] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:15:00.899256 14844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9428100e4a364805863419b22ca3c682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:00.902117 15100 ts_tablet_manager.cc:1888] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 08:15:00.902398 15100 log.cc:1196] T 0f6758eb87b047d29a060fcbfb775903 P 0cb18b8b10f944e39829faf16e061fd6: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/wals/0f6758eb87b047d29a060fcbfb775903
I20240503 08:15:00.906558 9667 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.907706 14832 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.912542 15092 ts_tablet_manager.cc:1888] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:15:00.915364 9378 catalog_manager.cc:4830] TS 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473): tablet 0f6758eb87b047d29a060fcbfb775903 (table rebalance_test_table_1 [id=a823dae3135f45e484931ad3ca59ef9d]) successfully deleted
I20240503 08:15:00.918239 15092 log.cc:1196] T dd41cbc6c69441c1af5c36c382cfe366 P 3a70e21b51a142bd8f7dc93fa58742fb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal/wals/dd41cbc6c69441c1af5c36c382cfe366
I20240503 08:15:00.919282 9378 catalog_manager.cc:4830] TS 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): tablet dd41cbc6c69441c1af5c36c382cfe366 (table rebalance_test_table_2 [id=8d1efeef13284616b16a57fba2f0caaf]) successfully deleted
I20240503 08:15:00.920406 14844 tablet_copy_client.cc:783] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:00.920699 14844 tablet_copy_client.cc:647] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:00.923149 14844 tablet_copy_client.cc:523] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:00.926375 14998 tablet_copy_client.cc:783] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: tablet copy: Starting download of 0 data blocks...
I20240503 08:15:00.926620 14998 tablet_copy_client.cc:647] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: tablet copy: Starting download of 1 WAL segments...
I20240503 08:15:00.931957 14998 tablet_copy_client.cc:523] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:15:00.936525 14844 tablet_bootstrap.cc:492] T 9428100e4a364805863419b22ca3c682 P fa60e870de844325bad4b9185eaee468: Bootstrap starting.
I20240503 08:15:00.939018 10058 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.941329 15120 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:15:00.942122 9993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.123.9:44619, user_credentials={real_user=slave}} blocked reactor thread for 43555.7us
I20240503 08:15:00.942822 10321 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.944348 9927 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:00.947554 15119 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.954670 14638 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:00.960258 14732 raft_consensus.cc:1050] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51: attempting to promote NON_VOTER b3d3436263d04c57b2d47cc9d766efa5 to VOTER
I20240503 08:15:00.961356 14732 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } }
I20240503 08:15:00.966832 15121 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:00.976589 9534 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:00.980476 11957 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:00.981400 9667 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:00.986841 9386 catalog_manager.cc:5484] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:00.989832 12224 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.002565 14998 tablet_bootstrap.cc:492] T 49a60d2ed0744f35be8054955dcf8aa0 P 06f1b17731834350a289de93abab8542: Bootstrap starting.
I20240503 08:15:01.011884 10188 consensus_queue.cc:237] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } }
I20240503 08:15:01.015224 10057 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.015317 9667 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.016026 14937 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.016505 14937 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.017125 9532 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.017140 9798 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.018283 14822 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.018663 14822 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.021937 14937 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:01.026950 9798 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:01.033361 9928 raft_consensus.cc:1203] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:15:01.034308 12224 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.034986 14822 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.035647 9379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9306edc6261425cb2c7bef9caf9135b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:15:01.036718 12224 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:01.042940 9532 consensus_queue.cc:237] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } }
I20240503 08:15:01.043855 12089 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:01.044456 14639 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:01.046808 9393 catalog_manager.cc:5484] T c9306edc6261425cb2c7bef9caf9135b P ca5fa388d5b84a0abac1c05275155dcf reported cstate change: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New cstate: current_term: 1 leader_uuid: "fa60e870de844325bad4b9185eaee468" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:01.047761 10191 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.047787 11957 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.048332 14941 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.048616 12224 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.048672 9928 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.049177 14941 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.050236 14941 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.050658 14453 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.052716 11938 tablet_service.cc:1514] Processing DeleteTablet for tablet c9306edc6261425cb2c7bef9caf9135b with delete_type TABLET_DATA_TOMBSTONED (TS 3a70e21b51a142bd8f7dc93fa58742fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57252
I20240503 08:15:01.053169 15092 tablet_replica.cc:330] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: stopping tablet replica
I20240503 08:15:01.053359 15092 raft_consensus.cc:2229] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:01.053599 15092 raft_consensus.cc:2258] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:01.054689 15092 ts_tablet_manager.cc:1875] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:01.055037 9667 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.055727 14947 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.056248 14947 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:01.056960 11957 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:01.065457 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fe97b80e1294c4791560daf991bfcdb with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:15:01.062920 9667 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:01.066519 15092 ts_tablet_manager.cc:1888] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:15:01.066718 15092 log.cc:1196] T c9306edc6261425cb2c7bef9caf9135b P 3a70e21b51a142bd8f7dc93fa58742fb: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-8/wal/wals/c9306edc6261425cb2c7bef9caf9135b
W20240503 08:15:01.067121 9731 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.24.123.62:44669, user_credentials={real_user=slave}} blocked reactor thread for 61669.8us
I20240503 08:15:01.067355 9926 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:01.065236 9386 catalog_manager.cc:5484] T 3fe97b80e1294c4791560daf991bfcdb P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:01.068861 10191 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:01.071251 9532 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEARNER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:01.071997 9663 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:01.087911 9378 catalog_manager.cc:4830] TS 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9:44619): tablet c9306edc6261425cb2c7bef9caf9135b (table rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]) successfully deleted
I20240503 08:15:01.087713 12222 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:01.102748 10058 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.105391 9667 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:01.107786 10191 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:01.135306 9795 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:01.136466 11957 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:01.108454 15050 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:01.118353 9927 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.132712 10057 raft_consensus.cc:1261] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.140100 14732 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:01.152662 10321 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.163767 9798 consensus_queue.cc:237] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } }
I20240503 08:15:01.169523 9667 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.170239 14638 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.173403 9534 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.176280 14638 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.185477 14732 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:01.186565 14947 consensus_queue.cc:1035] T 3fe97b80e1294c4791560daf991bfcdb P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.196000 11957 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.196741 14638 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.197443 10319 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:15:01.202121 10054 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:01.205109 9534 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:01.214149 10057 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.214859 14638 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.217626 15119 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.219811 9926 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.221082 14832 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.222681 15028 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:01.223292 15028 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:01.222997 9926 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.224136 10057 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.225929 15050 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:15:01.227615 14732 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.245329 10054 raft_consensus.cc:2932] T 3fe97b80e1294c4791560daf991bfcdb P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } } }
I20240503 08:15:01.246215 9666 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.249091 10320 raft_consensus.cc:1261] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.249395 10188 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.253223 9532 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.242934 9667 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.254912 9533 consensus_queue.cc:237] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } }
I20240503 08:15:01.255441 12089 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
W20240503 08:15:01.260380 9469 consensus_peers.cc:485] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 -> Peer fa60e870de844325bad4b9185eaee468 (127.24.123.6:37175): Couldn't send request to peer fa60e870de844325bad4b9185eaee468. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:15:01.260782 12090 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:15:01.261600 14941 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:01.263183 9795 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:15:01.263571 11957 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:15:01.263727 14453 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:01.264847 9927 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:15:01.265453 14941 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:01.265966 14947 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:01.264703 9393 catalog_manager.cc:5484] T f13c7cfe35b3403f87940a124e6d9cda P 06f1b17731834350a289de93abab8542 reported cstate change: config changed from index 9 to 10, VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) evicted. New cstate: current_term: 1 leader_uuid: "8de08e679a904ee48ba659861c730725" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.268227 9666 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:15:01.268846 14941 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:01.270044 14947 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:01.272984 10319 raft_consensus.cc:1261] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer fa60e870de844325bad4b9185eaee468: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:15:01.274423 9795 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 8de08e679a904ee48ba659861c730725 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:01.278625 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f13c7cfe35b3403f87940a124e6d9cda with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:15:01.281633 11957 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:01.285429 15060 consensus_queue.cc:1035] T c9306edc6261425cb2c7bef9caf9135b P fa60e870de844325bad4b9185eaee468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.285864 10301 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe97b80e1294c4791560daf991bfcdb with delete_type TABLET_DATA_TOMBSTONED (TS 0cb18b8b10f944e39829faf16e061fd6 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:57798
I20240503 08:15:01.286303 15100 tablet_replica.cc:330] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6: stopping tablet replica
I20240503 08:15:01.286506 15100 raft_consensus.cc:2229] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:01.286808 15100 raft_consensus.cc:2258] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:01.287851 9385 catalog_manager.cc:5484] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 reported cstate change: config changed from index 11 to 12, b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a1402a75453d4ccf992606dcce77dc51" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.292043 15100 ts_tablet_manager.cc:1875] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:01.295413 9533 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.295372 11958 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.299933 15119 consensus_queue.cc:1035] T f13c7cfe35b3403f87940a124e6d9cda P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:15:01.312366 11958 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.313792 12090 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:01.314384 9926 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P 06f1b17731834350a289de93abab8542 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:01.324571 15100 ts_tablet_manager.cc:1888] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 08:15:01.324512 9385 catalog_manager.cc:5484] T 1be499fcc2bf4716b16a5983950d3ed8 P 3a70e21b51a142bd8f7dc93fa58742fb reported cstate change: config changed from index 13 to 14, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New cstate: current_term: 1 leader_uuid: "b3d3436263d04c57b2d47cc9d766efa5" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:01.325103 9666 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:01.328955 10054 raft_consensus.cc:1261] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Refusing update from remote peer b3d3436263d04c57b2d47cc9d766efa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:15:01.330147 15100 log.cc:1196] T 3fe97b80e1294c4791560daf991bfcdb P 0cb18b8b10f944e39829faf16e061fd6: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/wals/3fe97b80e1294c4791560daf991bfcdb
I20240503 08:15:01.336287 10320 raft_consensus.cc:2932] T f13c7cfe35b3403f87940a124e6d9cda P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ca5fa388d5b84a0abac1c05275155dcf (127.24.123.8) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } }
I20240503 08:15:01.342000 15152 raft_consensus.cc:1050] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6: attempting to promote NON_VOTER fa60e870de844325bad4b9185eaee468 to VOTER
I20240503 08:15:01.343501 15152 consensus_queue.cc:237] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true } }
I20240503 08:15:01.347472 10321 raft_consensus.cc:2932] T c9306edc6261425cb2c7bef9caf9135b P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 3a70e21b51a142bd8f7dc93fa58742fb (127.24.123.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } }
I20240503 08:15:01.357448 14941 consensus_queue.cc:1035] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:15:01.360446 10322 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b3d3436263d04c57b2d47cc9d766efa5 (127.24.123.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { replace: true } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.375560 10058 raft_consensus.cc:2932] T 1be499fcc2bf4716b16a5983950d3ed8 P a1402a75453d4ccf992606dcce77dc51 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } attrs { promote: false } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: NON_VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: true } } }
I20240503 08:15:01.401171 10057 consensus_queue.cc:237] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } }
I20240503 08:15:01.405620 9666 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P d1eb5e69c6264a20a6e8297cc5deb8d1 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.405663 12089 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.406309 14639 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.406805 14639 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.409611 9378 catalog_manager.cc:4830] TS 0cb18b8b10f944e39829faf16e061fd6 (127.24.123.7:42473): tablet 3fe97b80e1294c4791560daf991bfcdb (table rebalance_test_table_3 [id=fcb51ccd98a2481297ab7d8f3c4a2273]) successfully deleted
I20240503 08:15:01.410115 10301 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be499fcc2bf4716b16a5983950d3ed8 with delete_type TABLET_DATA_TOMBSTONED (TS 0cb18b8b10f944e39829faf16e061fd6 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:57798
I20240503 08:15:01.410538 15100 tablet_replica.cc:330] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6: stopping tablet replica
I20240503 08:15:01.410576 9378 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1be499fcc2bf4716b16a5983950d3ed8 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:15:01.410739 15100 raft_consensus.cc:2229] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:01.411340 15100 raft_consensus.cc:2258] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:01.412590 15100 ts_tablet_manager.cc:1875] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:15:01.416694 11957 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 LEARNER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.417398 12222 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.417436 14725 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: NON_VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.420598 10188 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P fa60e870de844325bad4b9185eaee468 [term 1 LEARNER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.421557 14537 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.421811 15100 ts_tablet_manager.cc:1888] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240503 08:15:01.421939 15100 log.cc:1196] T 1be499fcc2bf4716b16a5983950d3ed8 P 0cb18b8b10f944e39829faf16e061fd6: Deleting WAL directory at /tmp/dist-test-taskEUWrla/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723866223888-25068-0/raft_consensus-itest-cluster/ts-6/wal/wals/1be499fcc2bf4716b16a5983950d3ed8
I20240503 08:15:01.421842 15156 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.422344 12089 raft_consensus.cc:1261] T e11a6197ccdd4e8d9a203efae4a94988 P 925c339b047b4cf5bae840c5a357d3de [term 1 FOLLOWER]: Refusing update from remote peer 0cb18b8b10f944e39829faf16e061fd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.424010 9533 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1be499fcc2bf4716b16a5983950d3ed8"
dest_uuid: "b3d3436263d04c57b2d47cc9d766efa5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51848
I20240503 08:15:01.424209 14537 consensus_queue.cc:1035] T e11a6197ccdd4e8d9a203efae4a94988 P 0cb18b8b10f944e39829faf16e061fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.424391 9533 raft_consensus.cc:595] T 1be499fcc2bf4716b16a5983950d3ed8 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:15:01.438803 9533 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.440284 11958 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.441632 10319 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P 0cb18b8b10f944e39829faf16e061fd6 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.441884 14832 raft_consensus.cc:1050] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725: attempting to promote NON_VOTER 06f1b17731834350a289de93abab8542 to VOTER
I20240503 08:15:01.444422 14832 consensus_queue.cc:237] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } attrs { replace: true } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } } peers { permanent_uuid: "8de08e679a904ee48ba659861c730725" member_type: VOTER last_known_addr { host: "127.24.123.3" port: 41805 } } peers { permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8" port: 43883 } attrs { promote: false } } peers { permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false } }
I20240503 08:15:01.445348 14732 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.447183 14732 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.447651 14732 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.448956 10188 raft_consensus.cc:1261] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer a1402a75453d4ccf992606dcce77dc51: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:15:01.449510 14732 consensus_queue.cc:1035] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:15:01.451400 9926 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P 06f1b17731834350a289de93abab8542 [term 1 LEARNER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:01.452040 15119 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f1b17731834350a289de93abab8542" member_type: VOTER last_known_addr { host: "127.24.123.4" port: 38565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:01.452421 14732 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P a1402a75453d4ccf992606dcce77dc51 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.454077 11958 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P 3a70e21b51a142bd8f7dc93fa58742fb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.454789 10188 raft_consensus.cc:2932] T 43f3607501a84050962dc116210c09cd P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 8de08e679a904ee48ba659861c730725 (127.24.123.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a1402a75453d4ccf992606dcce77dc51" member_type: VOTER last_known_addr { host: "127.24.123.5" port: 43643 } } peers { permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 } } peers { permanent_uuid: "0cb18b8b10f944e39829faf16e061fd6" member_type: VOTER last_known_addr { host: "127.24.123.7" port: 42473 } } peers { permanent_uuid: "d1eb5e69c6264a20a6e8297cc5deb8d1" member_type: VOTER last_known_addr { host: "127.24.123.2" port: 36123 } } peers { permanent_uuid: "3a70e21b51a142bd8f7dc93fa58742fb" member_type: VOTER last_known_addr { host: "127.24.123.9" port: 44619 } attrs { promote: false } } peers { permanent_uuid: "925c339b047b4cf5bae840c5a357d3de" member_type: VOTER last_known_addr { host: "127.24.123.10" port: 33439 } attrs { promote: false } } peers { permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 } attrs { promote: false } } }
I20240503 08:15:01.456353 9534 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P b3d3436263d04c57b2d47cc9d766efa5 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:01.457466 12224 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P ca5fa388d5b84a0abac1c05275155dcf [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:01.458951 10189 raft_consensus.cc:1261] T 34be73cc40264224ad798d7518c45ed1 P fa60e870de844325bad4b9185eaee468 [term 1 FOLLOWER]: Refusing update from remote peer 8de08e679a904ee48ba659861c730725: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:15:01.459621 15119 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60e870de844325bad4b9185eaee468" member_type: VOTER last_known_addr { host: "127.24.123.6" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:01.463871 15119 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d3436263d04c57b2d47cc9d766efa5" member_type: VOTER last_known_addr { host: "127.24.123.1" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:15:01.464536 15119 consensus_queue.cc:1035] T 34be73cc40264224ad798d7518c45ed1 P 8de08e679a904ee48ba659861c730725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5fa388d5b84a0abac1c05275155dcf" member_type: VOTER last_known_addr { host: "127.24.123.8"